2022-04-10T01:08:24.438 INFO:root:teuthology version: 1.1.0-a347b2ac 2022-04-10T01:08:24.439 DEBUG:teuthology.run:Teuthology command: teuthology --description rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} msgr-failures/few objectstore/bluestore-hybrid policy/simple supported-random-distro$/{centos_8}} --owner scheduled_mrfreezeex@teuthology --verbose --name mrfreezeex-2022-04-09_15:53:21-rbd-wip-mrfreezeex-multipeer-distro-default-smithi --archive /home/teuthworker/archive/mrfreezeex-2022-04-09_15:53:21-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784449 -- /home/teuthworker/archive/mrfreezeex-2022-04-09_15:53:21-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784449/orig.config.yaml 2022-04-10T01:08:24.499 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-04-10T01:08:24.585 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/mrfreezeex-2022-04-09_15:53:21-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784449 branch: wip-mrfreezeex-multipeer description: rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} msgr-failures/few objectstore/bluestore-hybrid policy/simple supported-random-distro$/{centos_8}} email: arthur.outhenin-chalandre@cern.ch first_in_suite: false job_id: '6784449' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each - desc: run four rbd-mirror daemons per cluster - desc: run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon name: mrfreezeex-2022-04-09_15:53:21-rbd-wip-mrfreezeex-multipeer-distro-default-smithi nuke-on-error: true openstack: - volumes: count: 3 size: 30 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-mrfreezeex-multipeer ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 rbd mirror image policy type: simple client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluefs allocator: hybrid bluestore allocator: hybrid bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 9b3739e792ef182b79445b926c7b7da461c05f52 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 9b3739e792ef182b79445b926c7b7da461c05f52 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-mrfreezeex-multipeer sha1: 9b3739e792ef182b79445b926c7b7da461c05f52 owner: scheduled_mrfreezeex@teuthology priority: 165 repo: git://git.ceph.com/ceph-ci.git roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 sha1: 9b3739e792ef182b79445b926c7b7da461c05f52 sleep_before_teardown: 0 suite: rbd suite_branch: wip-mrfreezeex-multipeer suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_9b3739e792ef182b79445b926c7b7da461c05f52/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 9b3739e792ef182b79445b926c7b7da461c05f52 targets: smithi005.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6a7IpOZPtZwIOOYNtsNKq1njcAjwzrK01ZhS9QRS+ifxFUZFe8h8edywJ1zqQAOZRNH+E+aKD6C1F+nyv2HdWrjIXqbyTlgyYxU/VS9RHc27G2rFDS9p7aogFlkCDXHKfx+BiYqnbGBmqtGKL0Os0Ur7m1uZ0hn34qmSiZS0XD4/wSTSwW6XwhkJCMxuMaoWzthqmojoClGBbutqSDhmOsLBOf3g+7PchS3HOPGYIUJ+rd9p2lkM+3YV76Lo/9Sj5TrYxONbGx8tZEnw+xkYwrGqTbxTAax8DX7Bws8jRXQeKekmgA1m+JzxmWB7vTIx2Nm9wo0iMLWQT53nbhx2Mcjg224BSYyYvQiJnPX46yv25nTVAzLnVXQ8lxTUXmacy2qwHRyOfe8BNYCisGmU5Z9suX/pFgnFEBbzf5wcFmW/KwsuTDa/A2nzRNRpsaWBv53ZA7jNXBXIYOYoaV3GFYLA2AdhvsGTMzpXYxa9f+QUxhJWFvLiDmBJk0kcwhCk= smithi043.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxbBDJqyiaOu9Bbtq4sC6enMBXqfpHtMB96N/mf2lA/aaKGlTMNpp2Zye0gQ9z4HygetquzjtuApbZg+ldNp1ZyvL7lcYvBos5rNC2+zTYmhk6+9bvHmSv/51wk67aheDavAy1QvcABfu4F/qTSGfqoWAXv+BZXYH2KjS8M7Rrp9YbBjcoGRsGwwvAzHLqcK806WaVGWOJCdfcpsKWVrL3R/MbUZRVhvsi8PvAeGzMAnpsYO5nyNH2Syla720zcmld++unoK06P4fWkF8hFOq/ooREjY6oH+R5zVohi1pDrEAviod6TLCvAhauFyhBgT7bcCvLk2P3/VvC7CkiX17A4fLZYnMZpQHfgsSLvlbrLFgF767L2r+/ekL114Zhxc4EolKhuZlEodiLAvt3LVgb3FcGq+J68W/ejuEY9nmZEl4pfa/jsq5LVTFoBwjcpJd40J8QOcPMteeniG7rH77A+ScsxTVCvAhiSiLpwxNfZjU6FkwakxUJq+3gEjzy21s= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - rbd-mirror: client: cluster1.client.mirror.0 thrash: true - rbd-mirror: client: cluster1.client.mirror.1 thrash: true - rbd-mirror: client: cluster1.client.mirror.2 thrash: true - rbd-mirror: client: cluster1.client.mirror.3 thrash: true - rbd-mirror: client: cluster2.client.mirror.0 thrash: true - rbd-mirror: client: cluster2.client.mirror.1 thrash: true - rbd-mirror: client: cluster2.client.mirror.2 thrash: true - rbd-mirror: client: cluster2.client.mirror.3 thrash: true - rbd-mirror-thrash: cluster: cluster1 - rbd-mirror-thrash: cluster: cluster2 - workunit: clients: cluster1.client.mirror: - rbd/rbd_mirror_stress.sh env: CEPH_ARGS: '' MIRROR_IMAGE_MODE: snapshot MIRROR_POOL_MODE: image RBD_IMAGE_FEATURES: layering,exclusive-lock RBD_MIRROR_INSTANCES: '4' RBD_MIRROR_USE_EXISTING_CLUSTER: '1' RBD_MIRROR_USE_RBD_MIRROR: '1' timeout: 6h teuthology_branch: master teuthology_sha1: a347b2ac5400a43ba76a678399b1deda7dae1a6c timestamp: 2022-04-09_15:53:21 tube: smithi user: mrfreezeex verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.22301 2022-04-10T01:08:24.586 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_9b3739e792ef182b79445b926c7b7da461c05f52/qa; will attempt to use it 2022-04-10T01:08:24.587 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_9b3739e792ef182b79445b926c7b7da461c05f52/qa/tasks 2022-04-10T01:08:24.587 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-04-10T01:08:24.589 INFO:teuthology.task.internal:Checking packages... 2022-04-10T01:08:24.611 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '9b3739e792ef182b79445b926c7b7da461c05f52' 2022-04-10T01:08:24.611 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-04-10T01:08:24.612 INFO:teuthology.packaging:ref: None 2022-04-10T01:08:24.612 INFO:teuthology.packaging:tag: None 2022-04-10T01:08:24.612 INFO:teuthology.packaging:branch: wip-mrfreezeex-multipeer 2022-04-10T01:08:24.613 INFO:teuthology.packaging:sha1: 9b3739e792ef182b79445b926c7b7da461c05f52 2022-04-10T01:08:24.613 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-mrfreezeex-multipeer 2022-04-10T01:08:24.788 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-11492.g9b3739e7 2022-04-10T01:08:24.789 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-04-10T01:08:24.798 INFO:teuthology.task.internal:no buildpackages task found 2022-04-10T01:08:24.799 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-04-10T01:08:24.834 INFO:teuthology.task.internal:Saving configuration 2022-04-10T01:08:24.855 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-04-10T01:08:24.870 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-04-10T01:08:24.895 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi005.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mrfreezeex-2022-04-09_15:53:21-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784449', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-04-10 01:03:56.012364', 'locked_by': 'scheduled_mrfreezeex@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ8OcWauJeYJLp4sXmDYz6X/4DUHKEpRKfFRUsZ0TGkxtjQ0jN/H2ud5lpqtE7jfM1Uw9mkEbg4msV8gp5bXLZc='} 2022-04-10T01:08:24.917 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi043.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mrfreezeex-2022-04-09_15:53:21-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784449', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-04-10 01:03:56.014279', 'locked_by': 'scheduled_mrfreezeex@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDbecRDFKCARoBCX1H+uIbhEjrSFoG8b4r9dEUpozGEYfQ2pVDZnYqhfPqjVWaVtewISqSGOQSpxp1UCb6hQYDg='} 2022-04-10T01:08:24.917 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-04-10T01:08:24.929 INFO:teuthology.task.internal:roles: ubuntu@smithi005.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2022-04-10T01:08:24.930 INFO:teuthology.task.internal:roles: ubuntu@smithi043.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2022-04-10T01:08:24.930 INFO:teuthology.run_tasks:Running task console_log... 2022-04-10T01:08:25.029 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f257b9e7a60>, signals=[15]) 2022-04-10T01:08:25.031 INFO:teuthology.run_tasks:Running task internal.connect... 2022-04-10T01:08:25.042 INFO:teuthology.task.internal:Opening connections... 2022-04-10T01:08:25.042 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi005.front.sepia.ceph.com 2022-04-10T01:08:25.044 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-04-10T01:08:25.114 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi043.front.sepia.ceph.com 2022-04-10T01:08:25.115 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-04-10T01:08:25.184 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-04-10T01:08:25.198 DEBUG:teuthology.orchestra.run.smithi005:> uname -m 2022-04-10T01:08:25.221 INFO:teuthology.orchestra.run.smithi005.stdout:x86_64 2022-04-10T01:08:25.221 DEBUG:teuthology.orchestra.run.smithi005:> cat /etc/os-release 2022-04-10T01:08:25.281 INFO:teuthology.orchestra.run.smithi005.stdout:NAME="CentOS Stream" 2022-04-10T01:08:25.281 INFO:teuthology.orchestra.run.smithi005.stdout:VERSION="8" 2022-04-10T01:08:25.281 INFO:teuthology.orchestra.run.smithi005.stdout:ID="centos" 2022-04-10T01:08:25.282 INFO:teuthology.orchestra.run.smithi005.stdout:ID_LIKE="rhel fedora" 2022-04-10T01:08:25.282 INFO:teuthology.orchestra.run.smithi005.stdout:VERSION_ID="8" 2022-04-10T01:08:25.282 INFO:teuthology.orchestra.run.smithi005.stdout:PLATFORM_ID="platform:el8" 2022-04-10T01:08:25.282 INFO:teuthology.orchestra.run.smithi005.stdout:PRETTY_NAME="CentOS Stream 8" 2022-04-10T01:08:25.283 INFO:teuthology.orchestra.run.smithi005.stdout:ANSI_COLOR="0;31" 2022-04-10T01:08:25.283 INFO:teuthology.orchestra.run.smithi005.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-04-10T01:08:25.283 INFO:teuthology.orchestra.run.smithi005.stdout:HOME_URL="https://centos.org/" 2022-04-10T01:08:25.283 INFO:teuthology.orchestra.run.smithi005.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-04-10T01:08:25.284 INFO:teuthology.orchestra.run.smithi005.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-04-10T01:08:25.284 INFO:teuthology.orchestra.run.smithi005.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-04-10T01:08:25.285 INFO:teuthology.lock.ops:Updating smithi005.front.sepia.ceph.com on lock server 2022-04-10T01:08:25.308 DEBUG:teuthology.orchestra.run.smithi043:> uname -m 2022-04-10T01:08:25.329 INFO:teuthology.orchestra.run.smithi043.stdout:x86_64 2022-04-10T01:08:25.330 DEBUG:teuthology.orchestra.run.smithi043:> cat /etc/os-release 2022-04-10T01:08:25.390 INFO:teuthology.orchestra.run.smithi043.stdout:NAME="CentOS Stream" 2022-04-10T01:08:25.390 INFO:teuthology.orchestra.run.smithi043.stdout:VERSION="8" 2022-04-10T01:08:25.390 INFO:teuthology.orchestra.run.smithi043.stdout:ID="centos" 2022-04-10T01:08:25.391 INFO:teuthology.orchestra.run.smithi043.stdout:ID_LIKE="rhel fedora" 2022-04-10T01:08:25.391 INFO:teuthology.orchestra.run.smithi043.stdout:VERSION_ID="8" 2022-04-10T01:08:25.391 INFO:teuthology.orchestra.run.smithi043.stdout:PLATFORM_ID="platform:el8" 2022-04-10T01:08:25.391 INFO:teuthology.orchestra.run.smithi043.stdout:PRETTY_NAME="CentOS Stream 8" 2022-04-10T01:08:25.392 INFO:teuthology.orchestra.run.smithi043.stdout:ANSI_COLOR="0;31" 2022-04-10T01:08:25.392 INFO:teuthology.orchestra.run.smithi043.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-04-10T01:08:25.392 INFO:teuthology.orchestra.run.smithi043.stdout:HOME_URL="https://centos.org/" 2022-04-10T01:08:25.392 INFO:teuthology.orchestra.run.smithi043.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-04-10T01:08:25.393 INFO:teuthology.orchestra.run.smithi043.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-04-10T01:08:25.393 INFO:teuthology.orchestra.run.smithi043.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-04-10T01:08:25.393 INFO:teuthology.lock.ops:Updating smithi043.front.sepia.ceph.com on lock server 2022-04-10T01:08:25.415 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-04-10T01:08:25.430 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-04-10T01:08:25.458 INFO:teuthology.task.internal:Checking for old test directory... 2022-04-10T01:08:25.459 DEBUG:teuthology.orchestra.run.smithi005:> test '!' -e /home/ubuntu/cephtest 2022-04-10T01:08:25.462 DEBUG:teuthology.orchestra.run.smithi043:> test '!' -e /home/ubuntu/cephtest 2022-04-10T01:08:25.482 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-04-10T01:08:25.494 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-04-10T01:08:25.495 DEBUG:teuthology.orchestra.run.smithi005:> test -z $(ls -A /var/lib/ceph) 2022-04-10T01:08:25.522 DEBUG:teuthology.orchestra.run.smithi043:> test -z $(ls -A /var/lib/ceph) 2022-04-10T01:08:25.609 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-04-10T01:08:25.668 INFO:teuthology.run_tasks:Running task kernel... 2022-04-10T01:08:25.687 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2022-04-10T01:08:25.688 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'cluster2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2022-04-10T01:08:25.688 DEBUG:teuthology.orchestra.run.smithi005:> test -f /run/.containerenv -o -f /.dockerenv 2022-04-10T01:08:25.689 DEBUG:teuthology.orchestra.run.smithi043:> test -f /run/.containerenv -o -f /.dockerenv 2022-04-10T01:08:25.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-04-10T01:08:25.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-04-10T01:08:25.711 DEBUG:teuthology.orchestra.run.smithi005:> uname -r 2022-04-10T01:08:25.711 DEBUG:teuthology.orchestra.run.smithi043:> uname -r 2022-04-10T01:08:25.771 INFO:teuthology.orchestra.run.smithi043.stdout:4.18.0-373.el8.x86_64 2022-04-10T01:08:25.771 INFO:teuthology.task.kernel:Running kernel on smithi043: 4.18.0-373.el8.x86_64 2022-04-10T01:08:25.772 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum install -y kernel 2022-04-10T01:08:25.773 INFO:teuthology.orchestra.run.smithi005.stdout:4.18.0-373.el8.x86_64 2022-04-10T01:08:25.773 INFO:teuthology.task.kernel:Running kernel on smithi005: 4.18.0-373.el8.x86_64 2022-04-10T01:08:25.774 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum install -y kernel 2022-04-10T01:08:28.594 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS-8 - AppStream 28 MB/s | 21 MB 00:00 2022-04-10T01:08:28.707 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS-8 - AppStream 28 MB/s | 21 MB 00:00 2022-04-10T01:08:33.113 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS-8 - Base 34 MB/s | 21 MB 00:00 2022-04-10T01:08:33.232 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS-8 - Base 37 MB/s | 21 MB 00:00 2022-04-10T01:08:36.420 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS-8 - Extras 169 kB/s | 18 kB 00:00 2022-04-10T01:08:36.681 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS-8 - Extras 114 kB/s | 18 kB 00:00 2022-04-10T01:08:36.823 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS-8 - PowerTools 17 MB/s | 4.5 MB 00:00 2022-04-10T01:08:37.067 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS-8 - PowerTools 17 MB/s | 4.5 MB 00:00 2022-04-10T01:08:37.888 INFO:teuthology.orchestra.run.smithi043.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2022-04-10T01:08:38.149 INFO:teuthology.orchestra.run.smithi005.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2022-04-10T01:08:39.339 INFO:teuthology.orchestra.run.smithi043.stdout:Extra Packages for Enterprise Linux 8.5 MB/s | 11 MB 00:01 2022-04-10T01:08:39.733 INFO:teuthology.orchestra.run.smithi005.stdout:Extra Packages for Enterprise Linux 7.8 MB/s | 11 MB 00:01 2022-04-10T01:08:42.280 INFO:teuthology.orchestra.run.smithi043.stdout:lab-extras 407 kB/s | 24 kB 00:00 2022-04-10T01:08:42.642 INFO:teuthology.orchestra.run.smithi005.stdout:lab-extras 280 kB/s | 24 kB 00:00 2022-04-10T01:08:44.986 INFO:teuthology.orchestra.run.smithi043.stdout:Package kernel-4.18.0-373.el8.x86_64 is already installed. 2022-04-10T01:08:44.986 INFO:teuthology.orchestra.run.smithi043.stdout:Package kernel-4.18.0-305.3.1.el8.x86_64 is already installed. 2022-04-10T01:08:45.088 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2022-04-10T01:08:45.090 INFO:teuthology.orchestra.run.smithi043.stdout:Nothing to do. 2022-04-10T01:08:45.090 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2022-04-10T01:08:45.215 DEBUG:teuthology.orchestra.run.smithi043:> echo no | sudo yum reinstall kernel || true 2022-04-10T01:08:45.264 INFO:teuthology.orchestra.run.smithi005.stdout:Package kernel-4.18.0-373.el8.x86_64 is already installed. 2022-04-10T01:08:45.264 INFO:teuthology.orchestra.run.smithi005.stdout:Package kernel-4.18.0-305.3.1.el8.x86_64 is already installed. 2022-04-10T01:08:45.358 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2022-04-10T01:08:45.360 INFO:teuthology.orchestra.run.smithi005.stdout:Nothing to do. 2022-04-10T01:08:45.360 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2022-04-10T01:08:45.514 DEBUG:teuthology.orchestra.run.smithi005:> echo no | sudo yum reinstall kernel || true 2022-04-10T01:08:45.794 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:00:03 ago on Sun 10 Apr 2022 01:08:42 AM UTC. 2022-04-10T01:08:46.090 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:00:04 ago on Sun 10 Apr 2022 01:08:42 AM UTC. 2022-04-10T01:08:46.686 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2022-04-10T01:08:46.687 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2022-04-10T01:08:46.687 INFO:teuthology.orchestra.run.smithi043.stdout: Package Architecture Version Repository Size 2022-04-10T01:08:46.687 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2022-04-10T01:08:46.688 INFO:teuthology.orchestra.run.smithi043.stdout:Reinstalling: 2022-04-10T01:08:46.688 INFO:teuthology.orchestra.run.smithi043.stdout: kernel x86_64 4.18.0-373.el8 CentOS-Base 8.0 M 2022-04-10T01:08:46.688 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:08:46.688 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2022-04-10T01:08:46.689 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2022-04-10T01:08:46.689 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:08:46.689 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 8.0 M 2022-04-10T01:08:46.689 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 0 2022-04-10T01:08:46.690 INFO:teuthology.orchestra.run.smithi043.stderr:Operation aborted. 2022-04-10T01:08:46.759 INFO:teuthology.orchestra.run.smithi043.stdout:Is this ok [y/N]: 2022-04-10T01:08:46.759 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum reinstall -y kernel || true 2022-04-10T01:08:46.998 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2022-04-10T01:08:47.000 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2022-04-10T01:08:47.000 INFO:teuthology.orchestra.run.smithi005.stdout: Package Architecture Version Repository Size 2022-04-10T01:08:47.000 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2022-04-10T01:08:47.001 INFO:teuthology.orchestra.run.smithi005.stdout:Reinstalling: 2022-04-10T01:08:47.001 INFO:teuthology.orchestra.run.smithi005.stdout: kernel x86_64 4.18.0-373.el8 CentOS-Base 8.0 M 2022-04-10T01:08:47.001 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:08:47.001 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2022-04-10T01:08:47.002 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2022-04-10T01:08:47.002 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:08:47.002 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 8.0 M 2022-04-10T01:08:47.003 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 0 2022-04-10T01:08:47.003 INFO:teuthology.orchestra.run.smithi005.stderr:Operation aborted. 2022-04-10T01:08:47.068 INFO:teuthology.orchestra.run.smithi005.stdout:Is this ok [y/N]: 2022-04-10T01:08:47.069 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum reinstall -y kernel || true 2022-04-10T01:08:47.330 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:00:05 ago on Sun 10 Apr 2022 01:08:42 AM UTC. 2022-04-10T01:08:47.638 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:00:05 ago on Sun 10 Apr 2022 01:08:42 AM UTC. 2022-04-10T01:08:48.239 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2022-04-10T01:08:48.240 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2022-04-10T01:08:48.241 INFO:teuthology.orchestra.run.smithi043.stdout: Package Architecture Version Repository Size 2022-04-10T01:08:48.241 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2022-04-10T01:08:48.241 INFO:teuthology.orchestra.run.smithi043.stdout:Reinstalling: 2022-04-10T01:08:48.241 INFO:teuthology.orchestra.run.smithi043.stdout: kernel x86_64 4.18.0-373.el8 CentOS-Base 8.0 M 2022-04-10T01:08:48.242 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:08:48.242 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2022-04-10T01:08:48.242 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2022-04-10T01:08:48.242 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:08:48.243 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 8.0 M 2022-04-10T01:08:48.243 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 0 2022-04-10T01:08:48.243 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2022-04-10T01:08:48.457 INFO:teuthology.orchestra.run.smithi043.stdout:kernel-4.18.0-373.el8.x86_64.rpm 37 MB/s | 8.0 MB 00:00 2022-04-10T01:08:48.457 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2022-04-10T01:08:48.458 INFO:teuthology.orchestra.run.smithi043.stdout:Total 37 MB/s | 8.0 MB 00:00 2022-04-10T01:08:48.528 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2022-04-10T01:08:48.530 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2022-04-10T01:08:48.530 INFO:teuthology.orchestra.run.smithi005.stdout: Package Architecture Version Repository Size 2022-04-10T01:08:48.531 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2022-04-10T01:08:48.531 INFO:teuthology.orchestra.run.smithi005.stdout:Reinstalling: 2022-04-10T01:08:48.532 INFO:teuthology.orchestra.run.smithi005.stdout: kernel x86_64 4.18.0-373.el8 CentOS-Base 8.0 M 2022-04-10T01:08:48.532 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:08:48.532 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2022-04-10T01:08:48.533 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2022-04-10T01:08:48.533 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:08:48.534 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 8.0 M 2022-04-10T01:08:48.534 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 0 2022-04-10T01:08:48.535 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2022-04-10T01:08:48.548 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2022-04-10T01:08:48.743 INFO:teuthology.orchestra.run.smithi005.stdout:kernel-4.18.0-373.el8.x86_64.rpm 38 MB/s | 8.0 MB 00:00 2022-04-10T01:08:48.744 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2022-04-10T01:08:48.744 INFO:teuthology.orchestra.run.smithi005.stdout:Total 37 MB/s | 8.0 MB 00:00 2022-04-10T01:08:48.835 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2022-04-10T01:08:48.926 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2022-04-10T01:08:48.926 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2022-04-10T01:08:49.080 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2022-04-10T01:08:49.085 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2022-04-10T01:08:49.306 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2022-04-10T01:08:49.306 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2022-04-10T01:08:49.362 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2022-04-10T01:08:49.457 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2022-04-10T01:08:49.462 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2022-04-10T01:08:49.540 INFO:teuthology.orchestra.run.smithi043.stdout: Reinstalling : kernel-4.18.0-373.el8.x86_64 1/2 2022-04-10T01:08:49.541 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : kernel-4.18.0-373.el8.x86_64 2/2 2022-04-10T01:08:49.758 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2022-04-10T01:08:49.947 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: kernel-4.18.0-373.el8.x86_64 2/2 2022-04-10T01:08:49.947 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : kernel-4.18.0-373.el8.x86_64 1/2 2022-04-10T01:08:49.968 INFO:teuthology.orchestra.run.smithi005.stdout: Reinstalling : kernel-4.18.0-373.el8.x86_64 1/2 2022-04-10T01:08:49.969 INFO:teuthology.orchestra.run.smithi005.stdout: Cleanup : kernel-4.18.0-373.el8.x86_64 2/2 2022-04-10T01:08:50.374 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: kernel-4.18.0-373.el8.x86_64 2/2 2022-04-10T01:08:50.375 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : kernel-4.18.0-373.el8.x86_64 1/2 2022-04-10T01:08:50.389 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : kernel-4.18.0-373.el8.x86_64 2/2 2022-04-10T01:08:50.389 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:08:50.389 INFO:teuthology.orchestra.run.smithi043.stdout:Reinstalled: 2022-04-10T01:08:50.389 INFO:teuthology.orchestra.run.smithi043.stdout: kernel-4.18.0-373.el8.x86_64 2022-04-10T01:08:50.390 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:08:50.390 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2022-04-10T01:08:50.477 DEBUG:teuthology.orchestra.run.smithi043:> rpm -q kernel --last | head -n 1 2022-04-10T01:08:50.644 INFO:teuthology.orchestra.run.smithi043.stdout:kernel-4.18.0-373.el8.x86_64 Sun 10 Apr 2022 01:08:49 AM UTC 2022-04-10T01:08:50.645 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-373.el8.x86_64 2022-04-10T01:08:50.646 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2022-04-10T01:08:50.646 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2022-04-10T01:08:50.646 DEBUG:teuthology.orchestra.run.smithi043:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-04-10T01:08:50.678 INFO:teuthology.orchestra.run.smithi043.stdout:ttyS1 2022-04-10T01:08:50.692 DEBUG:teuthology.parallel:result is None 2022-04-10T01:08:50.842 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : kernel-4.18.0-373.el8.x86_64 2/2 2022-04-10T01:08:50.842 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:08:50.843 INFO:teuthology.orchestra.run.smithi005.stdout:Reinstalled: 2022-04-10T01:08:50.843 INFO:teuthology.orchestra.run.smithi005.stdout: kernel-4.18.0-373.el8.x86_64 2022-04-10T01:08:50.843 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:08:50.843 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2022-04-10T01:08:50.930 DEBUG:teuthology.orchestra.run.smithi005:> rpm -q kernel --last | head -n 1 2022-04-10T01:08:51.055 INFO:teuthology.orchestra.run.smithi005.stdout:kernel-4.18.0-373.el8.x86_64 Sun 10 Apr 2022 01:08:49 AM UTC 2022-04-10T01:08:51.057 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-373.el8.x86_64 2022-04-10T01:08:51.058 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2022-04-10T01:08:51.058 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2022-04-10T01:08:51.058 DEBUG:teuthology.orchestra.run.smithi005:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-04-10T01:08:51.096 INFO:teuthology.orchestra.run.smithi005.stdout:ttyS1 2022-04-10T01:08:51.109 DEBUG:teuthology.parallel:result is None 2022-04-10T01:08:51.110 INFO:teuthology.run_tasks:Running task internal.base... 2022-04-10T01:08:51.122 INFO:teuthology.task.internal:Creating test directory... 2022-04-10T01:08:51.123 DEBUG:teuthology.orchestra.run.smithi005:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-04-10T01:08:51.152 DEBUG:teuthology.orchestra.run.smithi043:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-04-10T01:08:51.171 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-04-10T01:08:51.183 INFO:teuthology.run_tasks:Running task internal.archive... 2022-04-10T01:08:51.233 INFO:teuthology.task.internal:Creating archive directory... 2022-04-10T01:08:51.233 DEBUG:teuthology.orchestra.run.smithi005:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-04-10T01:08:51.236 DEBUG:teuthology.orchestra.run.smithi043:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-04-10T01:08:51.272 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-04-10T01:08:51.292 INFO:teuthology.task.internal:Enabling coredump saving... 2022-04-10T01:08:51.292 DEBUG:teuthology.orchestra.run.smithi005:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-04-10T01:08:51.303 DEBUG:teuthology.orchestra.run.smithi043:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-04-10T01:08:51.338 INFO:teuthology.orchestra.run.smithi005.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-04-10T01:08:51.348 INFO:teuthology.orchestra.run.smithi005.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-04-10T01:08:51.350 INFO:teuthology.orchestra.run.smithi043.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-04-10T01:08:51.362 INFO:teuthology.orchestra.run.smithi043.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-04-10T01:08:51.364 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-04-10T01:08:51.376 INFO:teuthology.task.internal:Configuring sudo... 2022-04-10T01:08:51.377 DEBUG:teuthology.orchestra.run.smithi005:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-04-10T01:08:51.392 DEBUG:teuthology.orchestra.run.smithi043:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-04-10T01:08:51.434 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-04-10T01:08:51.446 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-04-10T01:08:51.447 DEBUG:teuthology.orchestra.run.smithi005:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-04-10T01:08:51.459 DEBUG:teuthology.orchestra.run.smithi043:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-04-10T01:08:51.492 DEBUG:teuthology.orchestra.run.smithi005:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-04-10T01:08:51.555 DEBUG:teuthology.orchestra.run.smithi005:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2022-04-10T01:08:51.615 DEBUG:teuthology.orchestra.run.smithi005:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-04-10T01:08:51.696 DEBUG:teuthology.orchestra.run.smithi005:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2022-04-10T01:08:51.740 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-04-10T01:08:51.741 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-04-10T01:08:51.807 DEBUG:teuthology.orchestra.run.smithi043:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-04-10T01:08:51.853 DEBUG:teuthology.orchestra.run.smithi043:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2022-04-10T01:08:51.912 DEBUG:teuthology.orchestra.run.smithi043:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-04-10T01:08:51.996 DEBUG:teuthology.orchestra.run.smithi043:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2022-04-10T01:08:52.046 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-04-10T01:08:52.047 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-04-10T01:08:52.119 DEBUG:teuthology.orchestra.run.smithi005:> sudo service rsyslog restart 2022-04-10T01:08:52.121 DEBUG:teuthology.orchestra.run.smithi043:> sudo service rsyslog restart 2022-04-10T01:08:52.178 INFO:teuthology.orchestra.run.smithi005.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2022-04-10T01:08:52.236 INFO:teuthology.orchestra.run.smithi043.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2022-04-10T01:08:52.673 INFO:teuthology.run_tasks:Running task internal.timer... 2022-04-10T01:08:52.683 INFO:teuthology.task.internal:Starting timer... 2022-04-10T01:08:52.683 INFO:teuthology.run_tasks:Running task pcp... 2022-04-10T01:08:52.761 INFO:teuthology.run_tasks:Running task selinux... 2022-04-10T01:08:52.826 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2022-04-10T01:08:52.827 DEBUG:teuthology.orchestra.run.smithi005:> sudo service auditd rotate 2022-04-10T01:08:52.892 INFO:teuthology.orchestra.run.smithi005.stdout:Rotating logs: 2022-04-10T01:08:52.893 DEBUG:teuthology.orchestra.run.smithi043:> sudo service auditd rotate 2022-04-10T01:08:52.961 INFO:teuthology.orchestra.run.smithi043.stdout:Rotating logs: 2022-04-10T01:08:52.963 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-04-10T01:08:52.963 DEBUG:teuthology.orchestra.run.smithi005:> /usr/sbin/getenforce 2022-04-10T01:08:52.987 INFO:teuthology.orchestra.run.smithi005.stdout:Permissive 2022-04-10T01:08:52.987 DEBUG:teuthology.orchestra.run.smithi043:> /usr/sbin/getenforce 2022-04-10T01:08:53.013 INFO:teuthology.orchestra.run.smithi043.stdout:Permissive 2022-04-10T01:08:53.013 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi005.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi043.front.sepia.ceph.com': 'permissive'} 2022-04-10T01:08:53.013 DEBUG:teuthology.orchestra.run.smithi005:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\|scontext=system_u:system_r:logrotate_t:s0\)' 2022-04-10T01:08:53.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-04-10T01:08:53.057 DEBUG:teuthology.orchestra.run.smithi043:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\|scontext=system_u:system_r:logrotate_t:s0\)' 2022-04-10T01:08:53.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-04-10T01:08:53.086 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-04-10T01:08:53.087 DEBUG:teuthology.orchestra.run.smithi005:> sudo /usr/sbin/setenforce permissive 2022-04-10T01:08:53.125 DEBUG:teuthology.orchestra.run.smithi043:> sudo /usr/sbin/setenforce permissive 2022-04-10T01:08:53.155 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-04-10T01:08:53.167 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2022-04-10T01:08:53.174 INFO:teuthology.repo_utils:Fetching master from origin 2022-04-10T01:08:53.205 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2022-04-10T01:08:53.226 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-04-10T01:08:53.228 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi005.front.sepia.ceph.com,smithi043.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2022-04-10T01:12:58.844 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi005.front.sepia.ceph.com'), Remote(name='ubuntu@smithi043.front.sepia.ceph.com')] 2022-04-10T01:12:58.847 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi005.front.sepia.ceph.com' 2022-04-10T01:12:58.849 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-04-10T01:12:58.924 DEBUG:teuthology.orchestra.run.smithi005:> true 2022-04-10T01:12:58.994 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi043.front.sepia.ceph.com' 2022-04-10T01:12:58.995 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-04-10T01:12:59.065 DEBUG:teuthology.orchestra.run.smithi043:> true 2022-04-10T01:12:59.134 INFO:teuthology.run_tasks:Running task clock... 2022-04-10T01:12:59.151 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-04-10T01:12:59.151 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-04-10T01:12:59.153 DEBUG:teuthology.orchestra.run.smithi005:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-04-10T01:12:59.156 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-04-10T01:12:59.156 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-04-10T01:12:59.189 INFO:teuthology.orchestra.run.smithi005.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2022-04-10T01:12:59.205 INFO:teuthology.orchestra.run.smithi005.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2022-04-10T01:12:59.211 INFO:teuthology.orchestra.run.smithi043.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2022-04-10T01:12:59.227 INFO:teuthology.orchestra.run.smithi043.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2022-04-10T01:12:59.280 INFO:teuthology.orchestra.run.smithi005.stderr:sudo: ntpd: command not found 2022-04-10T01:12:59.290 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: ntpd: command not found 2022-04-10T01:12:59.293 INFO:teuthology.orchestra.run.smithi005.stdout:506 Cannot talk to daemon 2022-04-10T01:12:59.302 INFO:teuthology.orchestra.run.smithi043.stdout:506 Cannot talk to daemon 2022-04-10T01:12:59.309 INFO:teuthology.orchestra.run.smithi005.stderr:Failed to start ntp.service: Unit ntp.service not found. 2022-04-10T01:12:59.317 INFO:teuthology.orchestra.run.smithi043.stderr:Failed to start ntp.service: Unit ntp.service not found. 2022-04-10T01:12:59.323 INFO:teuthology.orchestra.run.smithi005.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2022-04-10T01:12:59.332 INFO:teuthology.orchestra.run.smithi043.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2022-04-10T01:12:59.401 INFO:teuthology.orchestra.run.smithi005.stderr:bash: ntpq: command not found 2022-04-10T01:12:59.403 INFO:teuthology.orchestra.run.smithi043.stderr:bash: ntpq: command not found 2022-04-10T01:12:59.405 INFO:teuthology.orchestra.run.smithi005.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2022-04-10T01:12:59.405 INFO:teuthology.orchestra.run.smithi005.stdout:=============================================================================== 2022-04-10T01:12:59.405 INFO:teuthology.orchestra.run.smithi005.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-04-10T01:12:59.405 INFO:teuthology.orchestra.run.smithi005.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-04-10T01:12:59.406 INFO:teuthology.orchestra.run.smithi005.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-04-10T01:12:59.406 INFO:teuthology.orchestra.run.smithi005.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-04-10T01:12:59.407 INFO:teuthology.orchestra.run.smithi043.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2022-04-10T01:12:59.407 INFO:teuthology.orchestra.run.smithi043.stdout:=============================================================================== 2022-04-10T01:12:59.407 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-04-10T01:12:59.407 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-04-10T01:12:59.408 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-04-10T01:12:59.408 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-04-10T01:12:59.410 INFO:teuthology.run_tasks:Running task install... 2022-04-10T01:12:59.449 DEBUG:teuthology.task.install:project ceph 2022-04-10T01:12:59.450 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '9b3739e792ef182b79445b926c7b7da461c05f52'}} 2022-04-10T01:12:59.450 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '9b3739e792ef182b79445b926c7b7da461c05f52'} 2022-04-10T01:12:59.451 INFO:teuthology.task.install:Using flavor: default 2022-04-10T01:12:59.457 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2022-04-10T01:12:59.458 INFO:teuthology.task.install:extra packages: ['rbd-mirror'] 2022-04-10T01:12:59.459 DEBUG:teuthology.orchestra.run.smithi005:> sudo lsb_release -is 2022-04-10T01:12:59.495 INFO:teuthology.orchestra.run.smithi005.stdout:CentOSStream 2022-04-10T01:12:59.497 DEBUG:teuthology.misc:System to be installed: CentOSStream 2022-04-10T01:12:59.497 DEBUG:teuthology.orchestra.run.smithi043:> sudo lsb_release -is 2022-04-10T01:12:59.498 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '9b3739e792ef182b79445b926c7b7da461c05f52', 'tag': None, 'wait_for_package': False} 2022-04-10T01:12:59.499 DEBUG:teuthology.orchestra.run.smithi005:> sudo lsb_release -is 2022-04-10T01:12:59.536 INFO:teuthology.orchestra.run.smithi043.stdout:CentOSStream 2022-04-10T01:12:59.538 DEBUG:teuthology.misc:System to be installed: CentOSStream 2022-04-10T01:12:59.538 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '9b3739e792ef182b79445b926c7b7da461c05f52', 'tag': None, 'wait_for_package': False} 2022-04-10T01:12:59.539 DEBUG:teuthology.orchestra.run.smithi043:> sudo lsb_release -is 2022-04-10T01:12:59.575 INFO:teuthology.orchestra.run.smithi005.stdout:CentOSStream 2022-04-10T01:12:59.576 DEBUG:teuthology.misc:System to be installed: CentOSStream 2022-04-10T01:12:59.577 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=9b3739e792ef182b79445b926c7b7da461c05f52 2022-04-10T01:12:59.616 INFO:teuthology.orchestra.run.smithi043.stdout:CentOSStream 2022-04-10T01:12:59.618 DEBUG:teuthology.misc:System to be installed: CentOSStream 2022-04-10T01:12:59.618 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=9b3739e792ef182b79445b926c7b7da461c05f52 2022-04-10T01:12:59.735 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-mrfreezeex-multipeer/9b3739e792ef182b79445b926c7b7da461c05f52/centos/8/flavors/default/ 2022-04-10T01:12:59.735 INFO:teuthology.task.install.rpm:Package version is 17.0.0-11492.g9b3739e7 2022-04-10T01:12:59.771 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-mrfreezeex-multipeer/9b3739e792ef182b79445b926c7b7da461c05f52/centos/8/flavors/default/ 2022-04-10T01:12:59.771 INFO:teuthology.task.install.rpm:Package version is 17.0.0-11492.g9b3739e7 2022-04-10T01:12:59.857 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-mrfreezeex-multipeer/9b3739e792ef182b79445b926c7b7da461c05f52/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-mrfreezeex-multipeer/9b3739e792ef182b79445b926c7b7da461c05f52/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-mrfreezeex-multipeer/9b3739e792ef182b79445b926c7b7da461c05f52/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2022-04-10T01:12:59.857 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-04-10T01:12:59.858 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/yum.repos.d/ceph.repo 2022-04-10T01:12:59.887 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2022-04-10T01:12:59.888 DEBUG:teuthology.orchestra.run.smithi005:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/9b3739e792ef182b79445b926c7b7da461c05f52/;g' /etc/yum.repos.d/ceph.repo ; fi 2022-04-10T01:12:59.906 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-mrfreezeex-multipeer/9b3739e792ef182b79445b926c7b7da461c05f52/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-mrfreezeex-multipeer/9b3739e792ef182b79445b926c7b7da461c05f52/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-mrfreezeex-multipeer/9b3739e792ef182b79445b926c7b7da461c05f52/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2022-04-10T01:12:59.906 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-04-10T01:12:59.906 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/yum.repos.d/ceph.repo 2022-04-10T01:12:59.935 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2022-04-10T01:12:59.936 DEBUG:teuthology.orchestra.run.smithi043:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/9b3739e792ef182b79445b926c7b7da461c05f52/;g' /etc/yum.repos.d/ceph.repo ; fi 2022-04-10T01:12:59.956 DEBUG:teuthology.orchestra.run.smithi005:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2022-04-10T01:13:00.005 DEBUG:teuthology.orchestra.run.smithi043:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2022-04-10T01:13:00.035 DEBUG:teuthology.orchestra.run.smithi005:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2022-04-10T01:13:00.083 DEBUG:teuthology.orchestra.run.smithi043:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2022-04-10T01:13:00.103 INFO:teuthology.orchestra.run.smithi005.stdout:check_obsoletes = 1 2022-04-10T01:13:00.105 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum clean all 2022-04-10T01:13:00.149 INFO:teuthology.orchestra.run.smithi043.stdout:check_obsoletes = 1 2022-04-10T01:13:00.151 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum clean all 2022-04-10T01:13:00.353 INFO:teuthology.orchestra.run.smithi005.stdout:55 files removed 2022-04-10T01:13:00.373 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-radosgw 2022-04-10T01:13:00.398 INFO:teuthology.orchestra.run.smithi043.stdout:55 files removed 2022-04-10T01:13:00.418 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph-radosgw 2022-04-10T01:13:01.189 INFO:teuthology.orchestra.run.smithi005.stdout:ceph packages for x86_64 166 kB/s | 78 kB 00:00 2022-04-10T01:13:01.269 INFO:teuthology.orchestra.run.smithi043.stdout:ceph packages for x86_64 155 kB/s | 78 kB 00:00 2022-04-10T01:13:01.623 INFO:teuthology.orchestra.run.smithi005.stdout:ceph noarch packages 23 kB/s | 9.1 kB 00:00 2022-04-10T01:13:01.677 INFO:teuthology.orchestra.run.smithi043.stdout:ceph noarch packages 25 kB/s | 9.1 kB 00:00 2022-04-10T01:13:01.957 INFO:teuthology.orchestra.run.smithi005.stdout:ceph source packages 4.6 kB/s | 1.4 kB 00:00 2022-04-10T01:13:01.986 INFO:teuthology.orchestra.run.smithi043.stdout:ceph source packages 5.0 kB/s | 1.4 kB 00:00 2022-04-10T01:13:02.609 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS-8 - AppStream 34 MB/s | 21 MB 00:00 2022-04-10T01:13:02.646 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS-8 - AppStream 33 MB/s | 21 MB 00:00 2022-04-10T01:13:07.185 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS-8 - Base 35 MB/s | 21 MB 00:00 2022-04-10T01:13:07.280 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS-8 - Base 34 MB/s | 21 MB 00:00 2022-04-10T01:13:10.487 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS-8 - Extras 139 kB/s | 18 kB 00:00 2022-04-10T01:13:10.587 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS-8 - Extras 164 kB/s | 18 kB 00:00 2022-04-10T01:13:10.869 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS-8 - PowerTools 18 MB/s | 4.5 MB 00:00 2022-04-10T01:13:10.973 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS-8 - PowerTools 17 MB/s | 4.5 MB 00:00 2022-04-10T01:13:11.910 INFO:teuthology.orchestra.run.smithi005.stdout:Copr repo for python3-asyncssh owned by ceph 30 kB/s | 3.5 kB 00:00 2022-04-10T01:13:12.023 INFO:teuthology.orchestra.run.smithi043.stdout:Copr repo for python3-asyncssh owned by ceph 28 kB/s | 3.5 kB 00:00 2022-04-10T01:13:13.028 INFO:teuthology.orchestra.run.smithi005.stdout:Extra Packages for Enterprise Linux 11 MB/s | 11 MB 00:00 2022-04-10T01:13:13.065 INFO:teuthology.orchestra.run.smithi043.stdout:Extra Packages for Enterprise Linux 12 MB/s | 11 MB 00:00 2022-04-10T01:13:15.886 INFO:teuthology.orchestra.run.smithi005.stdout:lab-extras 393 kB/s | 24 kB 00:00 2022-04-10T01:13:16.032 INFO:teuthology.orchestra.run.smithi043.stdout:lab-extras 228 kB/s | 24 kB 00:00 2022-04-10T01:13:18.865 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2022-04-10T01:13:18.868 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2022-04-10T01:13:18.869 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repository Size 2022-04-10T01:13:18.869 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2022-04-10T01:13:18.869 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2022-04-10T01:13:18.869 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-radosgw x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 16 M 2022-04-10T01:13:18.870 INFO:teuthology.orchestra.run.smithi005.stdout:Upgrading: 2022-04-10T01:13:18.870 INFO:teuthology.orchestra.run.smithi005.stdout: librados2 x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 3.4 M 2022-04-10T01:13:18.870 INFO:teuthology.orchestra.run.smithi005.stdout: librbd1 x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 3.9 M 2022-04-10T01:13:18.870 INFO:teuthology.orchestra.run.smithi005.stdout:Installing dependencies: 2022-04-10T01:13:18.871 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-base x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 5.3 M 2022-04-10T01:13:18.871 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-common x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 23 M 2022-04-10T01:13:18.871 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-selinux x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 24 k 2022-04-10T01:13:18.871 INFO:teuthology.orchestra.run.smithi005.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2022-04-10T01:13:18.871 INFO:teuthology.orchestra.run.smithi005.stdout: libcephfs2 x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 795 k 2022-04-10T01:13:18.872 INFO:teuthology.orchestra.run.smithi005.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2022-04-10T01:13:18.872 INFO:teuthology.orchestra.run.smithi005.stdout: librabbitmq x86_64 0.9.0-3.el8 CentOS-Base 47 k 2022-04-10T01:13:18.872 INFO:teuthology.orchestra.run.smithi005.stdout: libradosstriper1 x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 484 k 2022-04-10T01:13:18.872 INFO:teuthology.orchestra.run.smithi005.stdout: librdkafka x86_64 0.11.4-3.el8 CentOS-AppStream 354 k 2022-04-10T01:13:18.873 INFO:teuthology.orchestra.run.smithi005.stdout: librgw2 x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 5.3 M 2022-04-10T01:13:18.873 INFO:teuthology.orchestra.run.smithi005.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2022-04-10T01:13:18.873 INFO:teuthology.orchestra.run.smithi005.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2022-04-10T01:13:18.873 INFO:teuthology.orchestra.run.smithi005.stdout: python3-ceph-argparse 2022-04-10T01:13:18.873 INFO:teuthology.orchestra.run.smithi005.stdout: x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 45 k 2022-04-10T01:13:18.874 INFO:teuthology.orchestra.run.smithi005.stdout: python3-ceph-common x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 106 k 2022-04-10T01:13:18.874 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cephfs x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 179 k 2022-04-10T01:13:18.875 INFO:teuthology.orchestra.run.smithi005.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2022-04-10T01:13:18.875 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rados x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 367 k 2022-04-10T01:13:18.876 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rbd x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 345 k 2022-04-10T01:13:18.876 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rgw x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 109 k 2022-04-10T01:13:18.876 INFO:teuthology.orchestra.run.smithi005.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2022-04-10T01:13:18.876 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:13:18.877 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2022-04-10T01:13:18.877 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2022-04-10T01:13:18.877 INFO:teuthology.orchestra.run.smithi005.stdout:Install 21 Packages 2022-04-10T01:13:18.877 INFO:teuthology.orchestra.run.smithi005.stdout:Upgrade 2 Packages 2022-04-10T01:13:18.878 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:13:18.878 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 62 M 2022-04-10T01:13:18.879 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2022-04-10T01:13:18.892 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2022-04-10T01:13:18.895 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2022-04-10T01:13:18.895 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repository Size 2022-04-10T01:13:18.895 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2022-04-10T01:13:18.896 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2022-04-10T01:13:18.896 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-radosgw x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 16 M 2022-04-10T01:13:18.896 INFO:teuthology.orchestra.run.smithi043.stdout:Upgrading: 2022-04-10T01:13:18.897 INFO:teuthology.orchestra.run.smithi043.stdout: librados2 x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 3.4 M 2022-04-10T01:13:18.897 INFO:teuthology.orchestra.run.smithi043.stdout: librbd1 x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 3.9 M 2022-04-10T01:13:18.897 INFO:teuthology.orchestra.run.smithi043.stdout:Installing dependencies: 2022-04-10T01:13:18.897 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-base x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 5.3 M 2022-04-10T01:13:18.897 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-common x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 23 M 2022-04-10T01:13:18.898 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-selinux x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 24 k 2022-04-10T01:13:18.898 INFO:teuthology.orchestra.run.smithi043.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2022-04-10T01:13:18.898 INFO:teuthology.orchestra.run.smithi043.stdout: libcephfs2 x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 795 k 2022-04-10T01:13:18.898 INFO:teuthology.orchestra.run.smithi043.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2022-04-10T01:13:18.899 INFO:teuthology.orchestra.run.smithi043.stdout: librabbitmq x86_64 0.9.0-3.el8 CentOS-Base 47 k 2022-04-10T01:13:18.899 INFO:teuthology.orchestra.run.smithi043.stdout: libradosstriper1 x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 484 k 2022-04-10T01:13:18.899 INFO:teuthology.orchestra.run.smithi043.stdout: librdkafka x86_64 0.11.4-3.el8 CentOS-AppStream 354 k 2022-04-10T01:13:18.899 INFO:teuthology.orchestra.run.smithi043.stdout: librgw2 x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 5.3 M 2022-04-10T01:13:18.900 INFO:teuthology.orchestra.run.smithi043.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2022-04-10T01:13:18.900 INFO:teuthology.orchestra.run.smithi043.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2022-04-10T01:13:18.900 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ceph-argparse 2022-04-10T01:13:18.901 INFO:teuthology.orchestra.run.smithi043.stdout: x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 45 k 2022-04-10T01:13:18.901 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ceph-common x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 106 k 2022-04-10T01:13:18.901 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cephfs x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 179 k 2022-04-10T01:13:18.901 INFO:teuthology.orchestra.run.smithi043.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2022-04-10T01:13:18.902 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rados x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 367 k 2022-04-10T01:13:18.902 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rbd x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 345 k 2022-04-10T01:13:18.902 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rgw x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 109 k 2022-04-10T01:13:18.902 INFO:teuthology.orchestra.run.smithi043.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2022-04-10T01:13:18.903 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:13:18.903 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2022-04-10T01:13:18.903 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2022-04-10T01:13:18.903 INFO:teuthology.orchestra.run.smithi043.stdout:Install 21 Packages 2022-04-10T01:13:18.903 INFO:teuthology.orchestra.run.smithi043.stdout:Upgrade 2 Packages 2022-04-10T01:13:18.904 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:13:18.904 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 62 M 2022-04-10T01:13:18.905 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2022-04-10T01:13:20.120 INFO:teuthology.orchestra.run.smithi005.stdout:(1/23): ceph-base-17.0.0-11492.g9b3739e7.el8.x8 4.2 MB/s | 5.3 MB 00:01 2022-04-10T01:13:20.196 INFO:teuthology.orchestra.run.smithi005.stdout:(2/23): ceph-selinux-17.0.0-11492.g9b3739e7.el8 324 kB/s | 24 kB 00:00 2022-04-10T01:13:20.199 INFO:teuthology.orchestra.run.smithi043.stdout:(1/23): ceph-base-17.0.0-11492.g9b3739e7.el8.x8 4.1 MB/s | 5.3 MB 00:01 2022-04-10T01:13:20.291 INFO:teuthology.orchestra.run.smithi043.stdout:(2/23): ceph-selinux-17.0.0-11492.g9b3739e7.el8 266 kB/s | 24 kB 00:00 2022-04-10T01:13:20.355 INFO:teuthology.orchestra.run.smithi005.stdout:(3/23): libcephfs2-17.0.0-11492.g9b3739e7.el8.x 4.9 MB/s | 795 kB 00:00 2022-04-10T01:13:20.683 INFO:teuthology.orchestra.run.smithi043.stdout:(3/23): libcephfs2-17.0.0-11492.g9b3739e7.el8.x 2.0 MB/s | 795 kB 00:00 2022-04-10T01:13:20.729 INFO:teuthology.orchestra.run.smithi005.stdout:(4/23): libradosstriper1-17.0.0-11492.g9b3739e7 1.3 MB/s | 484 kB 00:00 2022-04-10T01:13:21.266 INFO:teuthology.orchestra.run.smithi043.stdout:(4/23): libradosstriper1-17.0.0-11492.g9b3739e7 830 kB/s | 484 kB 00:00 2022-04-10T01:13:22.180 INFO:teuthology.orchestra.run.smithi005.stdout:(5/23): librgw2-17.0.0-11492.g9b3739e7.el8.x86_ 3.7 MB/s | 5.3 MB 00:01 2022-04-10T01:13:22.361 INFO:teuthology.orchestra.run.smithi005.stdout:(6/23): ceph-radosgw-17.0.0-11492.g9b3739e7.el8 4.6 MB/s | 16 MB 00:03 2022-04-10T01:13:22.383 INFO:teuthology.orchestra.run.smithi043.stdout:(5/23): ceph-common-17.0.0-11492.g9b3739e7.el8. 6.6 MB/s | 23 MB 00:03 2022-04-10T01:13:22.388 INFO:teuthology.orchestra.run.smithi005.stdout:(7/23): python3-ceph-argparse-17.0.0-11492.g9b3 214 kB/s | 45 kB 00:00 2022-04-10T01:13:22.405 INFO:teuthology.orchestra.run.smithi005.stdout:(8/23): python3-ceph-common-17.0.0-11492.g9b373 2.3 MB/s | 106 kB 00:00 2022-04-10T01:13:22.448 INFO:teuthology.orchestra.run.smithi005.stdout:(9/23): python3-cephfs-17.0.0-11492.g9b3739e7.e 3.0 MB/s | 179 kB 00:00 2022-04-10T01:13:22.506 INFO:teuthology.orchestra.run.smithi005.stdout:(10/23): python3-rados-17.0.0-11492.g9b3739e7.e 3.6 MB/s | 367 kB 00:00 2022-04-10T01:13:22.565 INFO:teuthology.orchestra.run.smithi005.stdout:(11/23): python3-rbd-17.0.0-11492.g9b3739e7.el8 2.9 MB/s | 345 kB 00:00 2022-04-10T01:13:22.584 INFO:teuthology.orchestra.run.smithi043.stdout:(6/23): ceph-radosgw-17.0.0-11492.g9b3739e7.el8 4.4 MB/s | 16 MB 00:03 2022-04-10T01:13:22.590 INFO:teuthology.orchestra.run.smithi005.stdout:(12/23): python3-rgw-17.0.0-11492.g9b3739e7.el8 1.3 MB/s | 109 kB 00:00 2022-04-10T01:13:22.608 INFO:teuthology.orchestra.run.smithi043.stdout:(7/23): python3-ceph-argparse-17.0.0-11492.g9b3 198 kB/s | 45 kB 00:00 2022-04-10T01:13:22.634 INFO:teuthology.orchestra.run.smithi043.stdout:(8/23): python3-ceph-common-17.0.0-11492.g9b373 2.1 MB/s | 106 kB 00:00 2022-04-10T01:13:22.659 INFO:teuthology.orchestra.run.smithi043.stdout:(9/23): python3-cephfs-17.0.0-11492.g9b3739e7.e 3.5 MB/s | 179 kB 00:00 2022-04-10T01:13:22.680 INFO:teuthology.orchestra.run.smithi005.stdout:(13/23): librdkafka-0.11.4-3.el8.x86_64.rpm 3.0 MB/s | 354 kB 00:00 2022-04-10T01:13:22.699 INFO:teuthology.orchestra.run.smithi005.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 2.4 MB/s | 44 kB 00:00 2022-04-10T01:13:22.709 INFO:teuthology.orchestra.run.smithi043.stdout:(10/23): python3-rados-17.0.0-11492.g9b3739e7.e 4.8 MB/s | 367 kB 00:00 2022-04-10T01:13:22.724 INFO:teuthology.orchestra.run.smithi005.stdout:(15/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.9 MB/s | 259 kB 00:00 2022-04-10T01:13:22.735 INFO:teuthology.orchestra.run.smithi043.stdout:(11/23): python3-rbd-17.0.0-11492.g9b3739e7.el8 4.5 MB/s | 345 kB 00:00 2022-04-10T01:13:22.741 INFO:teuthology.orchestra.run.smithi005.stdout:(16/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2022-04-10T01:13:22.760 INFO:teuthology.orchestra.run.smithi043.stdout:(12/23): python3-rgw-17.0.0-11492.g9b3739e7.el8 2.1 MB/s | 109 kB 00:00 2022-04-10T01:13:22.819 INFO:teuthology.orchestra.run.smithi043.stdout:(13/23): librdkafka-0.11.4-3.el8.x86_64.rpm 4.1 MB/s | 354 kB 00:00 2022-04-10T01:13:22.844 INFO:teuthology.orchestra.run.smithi043.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 1.7 MB/s | 44 kB 00:00 2022-04-10T01:13:22.869 INFO:teuthology.orchestra.run.smithi043.stdout:(15/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 2.4 MB/s | 259 kB 00:00 2022-04-10T01:13:22.895 INFO:teuthology.orchestra.run.smithi043.stdout:(16/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 935 kB/s | 47 kB 00:00 2022-04-10T01:13:22.995 INFO:teuthology.orchestra.run.smithi043.stdout:(17/23): librgw2-17.0.0-11492.g9b3739e7.el8.x86 3.1 MB/s | 5.3 MB 00:01 2022-04-10T01:13:23.100 INFO:teuthology.orchestra.run.smithi005.stdout:(17/23): ceph-common-17.0.0-11492.g9b3739e7.el8 5.5 MB/s | 23 MB 00:04 2022-04-10T01:13:23.103 INFO:teuthology.orchestra.run.smithi043.stdout:(18/23): liboath-2.6.2-3.el8.x86_64.rpm 282 kB/s | 59 kB 00:00 2022-04-10T01:13:23.117 INFO:teuthology.orchestra.run.smithi005.stdout:(18/23): liboath-2.6.2-3.el8.x86_64.rpm 157 kB/s | 59 kB 00:00 2022-04-10T01:13:23.154 INFO:teuthology.orchestra.run.smithi043.stdout:(19/23): libunwind-1.3.1-3.el8.x86_64.rpm 474 kB/s | 75 kB 00:00 2022-04-10T01:13:23.179 INFO:teuthology.orchestra.run.smithi043.stdout:(20/23): gperftools-libs-2.7-9.el8.x86_64.rpm 990 kB/s | 306 kB 00:00 2022-04-10T01:13:23.183 INFO:teuthology.orchestra.run.smithi005.stdout:(19/23): gperftools-libs-2.7-9.el8.x86_64.rpm 667 kB/s | 306 kB 00:00 2022-04-10T01:13:23.259 INFO:teuthology.orchestra.run.smithi005.stdout:(20/23): libunwind-1.3.1-3.el8.x86_64.rpm 473 kB/s | 75 kB 00:00 2022-04-10T01:13:23.580 INFO:teuthology.orchestra.run.smithi043.stdout:(21/23): librbd1-17.0.0-11492.g9b3739e7.el8.x86 9.7 MB/s | 3.9 MB 00:00 2022-04-10T01:13:23.646 INFO:teuthology.orchestra.run.smithi043.stdout:(22/23): thrift-0.13.0-2.el8.x86_64.rpm 3.2 MB/s | 1.7 MB 00:00 2022-04-10T01:13:23.659 INFO:teuthology.orchestra.run.smithi005.stdout:(21/23): thrift-0.13.0-2.el8.x86_64.rpm 3.2 MB/s | 1.7 MB 00:00 2022-04-10T01:13:23.751 INFO:teuthology.orchestra.run.smithi005.stdout:(22/23): librados2-17.0.0-11492.g9b3739e7.el8.x 6.1 MB/s | 3.4 MB 00:00 2022-04-10T01:13:23.809 INFO:teuthology.orchestra.run.smithi005.stdout:(23/23): librbd1-17.0.0-11492.g9b3739e7.el8.x86 7.0 MB/s | 3.9 MB 00:00 2022-04-10T01:13:23.813 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2022-04-10T01:13:23.813 INFO:teuthology.orchestra.run.smithi005.stdout:Total 13 MB/s | 62 MB 00:04 2022-04-10T01:13:23.854 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2022-04-10T01:13:23.902 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2022-04-10T01:13:23.902 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2022-04-10T01:13:23.905 INFO:teuthology.orchestra.run.smithi043.stdout:(23/23): librados2-17.0.0-11492.g9b3739e7.el8.x 4.6 MB/s | 3.4 MB 00:00 2022-04-10T01:13:23.908 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2022-04-10T01:13:23.909 INFO:teuthology.orchestra.run.smithi043.stdout:Total 12 MB/s | 62 MB 00:05 2022-04-10T01:13:23.953 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2022-04-10T01:13:24.002 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2022-04-10T01:13:24.002 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2022-04-10T01:13:24.522 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2022-04-10T01:13:24.525 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2022-04-10T01:13:24.641 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2022-04-10T01:13:24.644 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2022-04-10T01:13:25.607 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2022-04-10T01:13:25.674 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2022-04-10T01:13:25.870 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2022-04-10T01:13:25.939 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2022-04-10T01:13:26.070 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2022-04-10T01:13:26.121 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2022-04-10T01:13:26.473 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2022-04-10T01:13:26.517 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2022-04-10T01:13:26.703 INFO:teuthology.orchestra.run.smithi005.stdout: Upgrading : librados2-2:17.0.0-11492.g9b3739e7.el8.x86_64 3/25 2022-04-10T01:13:26.712 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : librados2-2:17.0.0-11492.g9b3739e7.el8.x86_64 3/25 2022-04-10T01:13:26.798 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librados2-2:17.0.0-11492.g9b3739e7.el8.x86_64 3/25 2022-04-10T01:13:26.803 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: librados2-2:17.0.0-11492.g9b3739e7.el8.x86_64 3/25 2022-04-10T01:13:27.066 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-rados-2:17.0.0-11492.g9b3739e7.el8.x86_64 4/25 2022-04-10T01:13:27.067 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-rados-2:17.0.0-11492.g9b3739e7.el8.x86_64 4/25 2022-04-10T01:13:27.246 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libcephfs2-2:17.0.0-11492.g9b3739e7.el8.x86_64 5/25 2022-04-10T01:13:27.253 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libcephfs2-2:17.0.0-11492.g9b3739e7.el8.x86_64 5/25 2022-04-10T01:13:27.691 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: libcephfs2-2:17.0.0-11492.g9b3739e7.el8.x86_64 5/25 2022-04-10T01:13:27.695 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: libcephfs2-2:17.0.0-11492.g9b3739e7.el8.x86_64 5/25 2022-04-10T01:13:27.904 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : librbd1-2:17.0.0-11492.g9b3739e7.el8.x86_64 6/25 2022-04-10T01:13:27.911 INFO:teuthology.orchestra.run.smithi005.stdout: Upgrading : librbd1-2:17.0.0-11492.g9b3739e7.el8.x86_64 6/25 2022-04-10T01:13:28.133 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: librbd1-2:17.0.0-11492.g9b3739e7.el8.x86_64 6/25 2022-04-10T01:13:28.138 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librbd1-2:17.0.0-11492.g9b3739e7.el8.x86_64 6/25 2022-04-10T01:13:28.358 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2022-04-10T01:13:28.384 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2022-04-10T01:13:28.579 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/25 2022-04-10T01:13:28.611 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/25 2022-04-10T01:13:28.729 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2022-04-10T01:13:28.802 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2022-04-10T01:13:29.302 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2022-04-10T01:13:29.373 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2022-04-10T01:13:29.479 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : librgw2-2:17.0.0-11492.g9b3739e7.el8.x86_64 10/25 2022-04-10T01:13:29.533 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librgw2-2:17.0.0-11492.g9b3739e7.el8.x86_64 10/25 2022-04-10T01:13:29.586 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : librgw2-2:17.0.0-11492.g9b3739e7.el8.x86_64 10/25 2022-04-10T01:13:29.656 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: librgw2-2:17.0.0-11492.g9b3739e7.el8.x86_64 10/25 2022-04-10T01:13:29.681 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-ceph-argparse-2:17.0.0-11492.g9b3739e7.el8 11/25 2022-04-10T01:13:29.832 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-ceph-argparse-2:17.0.0-11492.g9b3739e7.el8 11/25 2022-04-10T01:13:29.916 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cephfs-2:17.0.0-11492.g9b3739e7.el8.x86_64 12/25 2022-04-10T01:13:30.037 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-cephfs-2:17.0.0-11492.g9b3739e7.el8.x86_64 12/25 2022-04-10T01:13:30.148 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-rgw-2:17.0.0-11492.g9b3739e7.el8.x86_64 13/25 2022-04-10T01:13:30.264 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-rgw-2:17.0.0-11492.g9b3739e7.el8.x86_64 13/25 2022-04-10T01:13:30.426 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-rbd-2:17.0.0-11492.g9b3739e7.el8.x86_64 14/25 2022-04-10T01:13:30.533 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-rbd-2:17.0.0-11492.g9b3739e7.el8.x86_64 14/25 2022-04-10T01:13:30.604 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libradosstriper1-2:17.0.0-11492.g9b3739e7.el8.x86_ 15/25 2022-04-10T01:13:30.659 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: libradosstriper1-2:17.0.0-11492.g9b3739e7.el8.x86_ 15/25 2022-04-10T01:13:30.744 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libradosstriper1-2:17.0.0-11492.g9b3739e7.el8.x86_ 15/25 2022-04-10T01:13:30.806 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: libradosstriper1-2:17.0.0-11492.g9b3739e7.el8.x86_ 15/25 2022-04-10T01:13:30.903 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2022-04-10T01:13:31.077 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2022-04-10T01:13:31.097 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2022-04-10T01:13:31.294 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2022-04-10T01:13:31.320 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2022-04-10T01:13:31.543 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2022-04-10T01:13:31.562 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-ceph-common-2:17.0.0-11492.g9b3739e7.el8.x 19/25 2022-04-10T01:13:31.769 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-ceph-common-2:17.0.0-11492.g9b3739e7.el8.x 19/25 2022-04-10T01:13:35.401 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-common-2:17.0.0-11492.g9b3739e7.el8.x86_64 20/25 2022-04-10T01:13:35.579 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-common-2:17.0.0-11492.g9b3739e7.el8.x86_64 20/25 2022-04-10T01:13:35.595 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-common-2:17.0.0-11492.g9b3739e7.el8.x86_64 20/25 2022-04-10T01:13:35.819 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-common-2:17.0.0-11492.g9b3739e7.el8.x86_64 20/25 2022-04-10T01:13:36.205 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-common-2:17.0.0-11492.g9b3739e7.el8.x86_64 20/25 2022-04-10T01:13:36.441 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-common-2:17.0.0-11492.g9b3739e7.el8.x86_64 20/25 2022-04-10T01:13:36.509 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-base-2:17.0.0-11492.g9b3739e7.el8.x86_64 21/25 2022-04-10T01:13:36.667 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-base-2:17.0.0-11492.g9b3739e7.el8.x86_64 21/25 2022-04-10T01:13:36.700 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-base-2:17.0.0-11492.g9b3739e7.el8.x86_64 21/25 2022-04-10T01:13:36.744 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-base-2:17.0.0-11492.g9b3739e7.el8.x86_64 21/25 2022-04-10T01:13:36.894 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-selinux-2:17.0.0-11492.g9b3739e7.el8.x86_64 22/25 2022-04-10T01:13:36.943 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-selinux-2:17.0.0-11492.g9b3739e7.el8.x86_64 22/25 2022-04-10T01:13:56.203 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-selinux-2:17.0.0-11492.g9b3739e7.el8.x86_64 22/25 2022-04-10T01:13:56.204 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /sys 2022-04-10T01:13:56.204 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /proc 2022-04-10T01:13:56.205 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /dev 2022-04-10T01:13:56.205 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /run 2022-04-10T01:13:56.205 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /mnt 2022-04-10T01:13:56.206 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /var/tmp 2022-04-10T01:13:56.206 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /home 2022-04-10T01:13:56.207 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /tmp 2022-04-10T01:13:56.207 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /dev 2022-04-10T01:13:56.208 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:13:56.326 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-selinux-2:17.0.0-11492.g9b3739e7.el8.x86_64 22/25 2022-04-10T01:13:56.327 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /sys 2022-04-10T01:13:56.327 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /proc 2022-04-10T01:13:56.327 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /dev 2022-04-10T01:13:56.327 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /run 2022-04-10T01:13:56.328 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /mnt 2022-04-10T01:13:56.328 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /var/tmp 2022-04-10T01:13:56.328 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /home 2022-04-10T01:13:56.328 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /tmp 2022-04-10T01:13:56.329 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /dev 2022-04-10T01:13:56.329 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:13:57.981 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-radosgw-2:17.0.0-11492.g9b3739e7.el8.x86_64 23/25 2022-04-10T01:13:58.045 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-11492.g9b3739e7.el8.x86_64 23/25 2022-04-10T01:13:58.045 INFO:teuthology.orchestra.run.smithi005.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2022-04-10T01:13:58.081 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-radosgw-2:17.0.0-11492.g9b3739e7.el8.x86_64 23/25 2022-04-10T01:13:58.186 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-11492.g9b3739e7.el8.x86_64 23/25 2022-04-10T01:13:58.186 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2022-04-10T01:13:58.293 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2022-04-10T01:13:58.293 INFO:teuthology.orchestra.run.smithi005.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2022-04-10T01:13:58.386 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2022-04-10T01:13:58.386 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2022-04-10T01:13:58.947 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2022-04-10T01:13:58.947 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-base-2:17.0.0-11492.g9b3739e7.el8.x86_64 1/25 2022-04-10T01:13:58.947 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-common-2:17.0.0-11492.g9b3739e7.el8.x86_64 2/25 2022-04-10T01:13:58.948 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-radosgw-2:17.0.0-11492.g9b3739e7.el8.x86_64 3/25 2022-04-10T01:13:58.948 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-selinux-2:17.0.0-11492.g9b3739e7.el8.x86_64 4/25 2022-04-10T01:13:58.948 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libcephfs2-2:17.0.0-11492.g9b3739e7.el8.x86_64 5/25 2022-04-10T01:13:58.948 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libradosstriper1-2:17.0.0-11492.g9b3739e7.el8.x86_ 6/25 2022-04-10T01:13:58.949 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librgw2-2:17.0.0-11492.g9b3739e7.el8.x86_64 7/25 2022-04-10T01:13:58.949 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-ceph-argparse-2:17.0.0-11492.g9b3739e7.el8 8/25 2022-04-10T01:13:58.949 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-ceph-common-2:17.0.0-11492.g9b3739e7.el8.x 9/25 2022-04-10T01:13:58.949 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-cephfs-2:17.0.0-11492.g9b3739e7.el8.x86_64 10/25 2022-04-10T01:13:58.949 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-rados-2:17.0.0-11492.g9b3739e7.el8.x86_64 11/25 2022-04-10T01:13:58.950 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-rbd-2:17.0.0-11492.g9b3739e7.el8.x86_64 12/25 2022-04-10T01:13:58.950 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-rgw-2:17.0.0-11492.g9b3739e7.el8.x86_64 13/25 2022-04-10T01:13:58.950 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2022-04-10T01:13:58.950 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2022-04-10T01:13:58.951 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2022-04-10T01:13:58.951 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 17/25 2022-04-10T01:13:58.951 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2022-04-10T01:13:58.951 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2022-04-10T01:13:58.951 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2022-04-10T01:13:58.952 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2022-04-10T01:13:58.952 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librados2-2:17.0.0-11492.g9b3739e7.el8.x86_64 22/25 2022-04-10T01:13:58.952 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2022-04-10T01:13:58.952 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librbd1-2:17.0.0-11492.g9b3739e7.el8.x86_64 24/25 2022-04-10T01:13:59.050 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2022-04-10T01:13:59.050 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-base-2:17.0.0-11492.g9b3739e7.el8.x86_64 1/25 2022-04-10T01:13:59.050 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-common-2:17.0.0-11492.g9b3739e7.el8.x86_64 2/25 2022-04-10T01:13:59.050 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-radosgw-2:17.0.0-11492.g9b3739e7.el8.x86_64 3/25 2022-04-10T01:13:59.051 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-selinux-2:17.0.0-11492.g9b3739e7.el8.x86_64 4/25 2022-04-10T01:13:59.051 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libcephfs2-2:17.0.0-11492.g9b3739e7.el8.x86_64 5/25 2022-04-10T01:13:59.051 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libradosstriper1-2:17.0.0-11492.g9b3739e7.el8.x86_ 6/25 2022-04-10T01:13:59.051 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librgw2-2:17.0.0-11492.g9b3739e7.el8.x86_64 7/25 2022-04-10T01:13:59.052 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-ceph-argparse-2:17.0.0-11492.g9b3739e7.el8 8/25 2022-04-10T01:13:59.052 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-ceph-common-2:17.0.0-11492.g9b3739e7.el8.x 9/25 2022-04-10T01:13:59.052 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cephfs-2:17.0.0-11492.g9b3739e7.el8.x86_64 10/25 2022-04-10T01:13:59.052 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-rados-2:17.0.0-11492.g9b3739e7.el8.x86_64 11/25 2022-04-10T01:13:59.052 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-rbd-2:17.0.0-11492.g9b3739e7.el8.x86_64 12/25 2022-04-10T01:13:59.053 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-rgw-2:17.0.0-11492.g9b3739e7.el8.x86_64 13/25 2022-04-10T01:13:59.053 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2022-04-10T01:13:59.053 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2022-04-10T01:13:59.053 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2022-04-10T01:13:59.054 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 17/25 2022-04-10T01:13:59.054 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2022-04-10T01:13:59.054 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2022-04-10T01:13:59.054 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2022-04-10T01:13:59.055 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2022-04-10T01:13:59.055 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librados2-2:17.0.0-11492.g9b3739e7.el8.x86_64 22/25 2022-04-10T01:13:59.055 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2022-04-10T01:13:59.055 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librbd1-2:17.0.0-11492.g9b3739e7.el8.x86_64 24/25 2022-04-10T01:13:59.608 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2022-04-10T01:13:59.609 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:13:59.609 INFO:teuthology.orchestra.run.smithi005.stdout:Upgraded: 2022-04-10T01:13:59.609 INFO:teuthology.orchestra.run.smithi005.stdout: librados2-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:13:59.609 INFO:teuthology.orchestra.run.smithi005.stdout: librbd1-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:13:59.610 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2022-04-10T01:13:59.610 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-base-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:13:59.610 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-common-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:13:59.610 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-radosgw-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:13:59.611 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-selinux-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:13:59.611 INFO:teuthology.orchestra.run.smithi005.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2022-04-10T01:13:59.611 INFO:teuthology.orchestra.run.smithi005.stdout: libcephfs2-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:13:59.611 INFO:teuthology.orchestra.run.smithi005.stdout: liboath-2.6.2-3.el8.x86_64 2022-04-10T01:13:59.612 INFO:teuthology.orchestra.run.smithi005.stdout: librabbitmq-0.9.0-3.el8.x86_64 2022-04-10T01:13:59.612 INFO:teuthology.orchestra.run.smithi005.stdout: libradosstriper1-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:13:59.612 INFO:teuthology.orchestra.run.smithi005.stdout: librdkafka-0.11.4-3.el8.x86_64 2022-04-10T01:13:59.612 INFO:teuthology.orchestra.run.smithi005.stdout: librgw2-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:13:59.613 INFO:teuthology.orchestra.run.smithi005.stdout: libunwind-1.3.1-3.el8.x86_64 2022-04-10T01:13:59.613 INFO:teuthology.orchestra.run.smithi005.stdout: lttng-ust-2.8.1-11.el8.x86_64 2022-04-10T01:13:59.613 INFO:teuthology.orchestra.run.smithi005.stdout: python3-ceph-argparse-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:13:59.613 INFO:teuthology.orchestra.run.smithi005.stdout: python3-ceph-common-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:13:59.613 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cephfs-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:13:59.614 INFO:teuthology.orchestra.run.smithi005.stdout: python3-prettytable-0.7.2-14.el8.noarch 2022-04-10T01:13:59.614 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rados-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:13:59.614 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rbd-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:13:59.614 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rgw-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:13:59.615 INFO:teuthology.orchestra.run.smithi005.stdout: thrift-0.13.0-2.el8.x86_64 2022-04-10T01:13:59.615 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:13:59.615 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2022-04-10T01:13:59.748 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2022-04-10T01:13:59.748 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:13:59.749 INFO:teuthology.orchestra.run.smithi043.stdout:Upgraded: 2022-04-10T01:13:59.749 INFO:teuthology.orchestra.run.smithi043.stdout: librados2-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:13:59.749 INFO:teuthology.orchestra.run.smithi043.stdout: librbd1-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:13:59.749 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2022-04-10T01:13:59.750 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-base-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:13:59.750 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-common-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:13:59.750 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-radosgw-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:13:59.750 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-selinux-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:13:59.751 INFO:teuthology.orchestra.run.smithi043.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2022-04-10T01:13:59.751 INFO:teuthology.orchestra.run.smithi043.stdout: libcephfs2-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:13:59.751 INFO:teuthology.orchestra.run.smithi043.stdout: liboath-2.6.2-3.el8.x86_64 2022-04-10T01:13:59.751 INFO:teuthology.orchestra.run.smithi043.stdout: librabbitmq-0.9.0-3.el8.x86_64 2022-04-10T01:13:59.752 INFO:teuthology.orchestra.run.smithi043.stdout: libradosstriper1-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:13:59.752 INFO:teuthology.orchestra.run.smithi043.stdout: librdkafka-0.11.4-3.el8.x86_64 2022-04-10T01:13:59.752 INFO:teuthology.orchestra.run.smithi043.stdout: librgw2-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:13:59.752 INFO:teuthology.orchestra.run.smithi043.stdout: libunwind-1.3.1-3.el8.x86_64 2022-04-10T01:13:59.753 INFO:teuthology.orchestra.run.smithi043.stdout: lttng-ust-2.8.1-11.el8.x86_64 2022-04-10T01:13:59.753 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ceph-argparse-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:13:59.753 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ceph-common-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:13:59.754 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cephfs-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:13:59.754 INFO:teuthology.orchestra.run.smithi043.stdout: python3-prettytable-0.7.2-14.el8.noarch 2022-04-10T01:13:59.755 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rados-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:13:59.755 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rbd-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:13:59.755 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rgw-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:13:59.755 INFO:teuthology.orchestra.run.smithi043.stdout: thrift-0.13.0-2.el8.x86_64 2022-04-10T01:13:59.756 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:13:59.756 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2022-04-10T01:13:59.812 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-test 2022-04-10T01:13:59.924 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph-test 2022-04-10T01:14:00.443 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:00:45 ago on Sun 10 Apr 2022 01:13:15 AM UTC. 2022-04-10T01:14:00.566 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:00:45 ago on Sun 10 Apr 2022 01:13:15 AM UTC. 2022-04-10T01:14:01.330 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2022-04-10T01:14:01.331 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2022-04-10T01:14:01.331 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repository Size 2022-04-10T01:14:01.332 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2022-04-10T01:14:01.332 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2022-04-10T01:14:01.333 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-test x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 52 M 2022-04-10T01:14:01.333 INFO:teuthology.orchestra.run.smithi005.stdout:Installing dependencies: 2022-04-10T01:14:01.334 INFO:teuthology.orchestra.run.smithi005.stdout: jq x86_64 1.6-3.el8 CentOS-AppStream 202 k 2022-04-10T01:14:01.334 INFO:teuthology.orchestra.run.smithi005.stdout: libcephsqlite x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 170 k 2022-04-10T01:14:01.335 INFO:teuthology.orchestra.run.smithi005.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2022-04-10T01:14:01.335 INFO:teuthology.orchestra.run.smithi005.stdout: socat x86_64 1.7.4.1-1.el8 CentOS-AppStream 323 k 2022-04-10T01:14:01.335 INFO:teuthology.orchestra.run.smithi005.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2022-04-10T01:14:01.336 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:14:01.337 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2022-04-10T01:14:01.337 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2022-04-10T01:14:01.337 INFO:teuthology.orchestra.run.smithi005.stdout:Install 6 Packages 2022-04-10T01:14:01.338 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:14:01.339 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 53 M 2022-04-10T01:14:01.340 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 237 M 2022-04-10T01:14:01.340 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2022-04-10T01:14:01.399 INFO:teuthology.orchestra.run.smithi005.stdout:(1/6): jq-1.6-3.el8.x86_64.rpm 3.0 MB/s | 202 kB 00:00 2022-04-10T01:14:01.458 INFO:teuthology.orchestra.run.smithi005.stdout:(2/6): oniguruma-6.8.2-2.el8.x86_64.rpm 3.2 MB/s | 187 kB 00:00 2022-04-10T01:14:01.464 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2022-04-10T01:14:01.465 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2022-04-10T01:14:01.465 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repository Size 2022-04-10T01:14:01.465 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2022-04-10T01:14:01.466 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2022-04-10T01:14:01.466 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-test x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 52 M 2022-04-10T01:14:01.466 INFO:teuthology.orchestra.run.smithi043.stdout:Installing dependencies: 2022-04-10T01:14:01.466 INFO:teuthology.orchestra.run.smithi043.stdout: jq x86_64 1.6-3.el8 CentOS-AppStream 202 k 2022-04-10T01:14:01.467 INFO:teuthology.orchestra.run.smithi043.stdout: libcephsqlite x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 170 k 2022-04-10T01:14:01.467 INFO:teuthology.orchestra.run.smithi043.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2022-04-10T01:14:01.467 INFO:teuthology.orchestra.run.smithi043.stdout: socat x86_64 1.7.4.1-1.el8 CentOS-AppStream 323 k 2022-04-10T01:14:01.467 INFO:teuthology.orchestra.run.smithi043.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2022-04-10T01:14:01.468 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:14:01.468 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2022-04-10T01:14:01.468 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2022-04-10T01:14:01.468 INFO:teuthology.orchestra.run.smithi043.stdout:Install 6 Packages 2022-04-10T01:14:01.468 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:14:01.469 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 53 M 2022-04-10T01:14:01.470 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 237 M 2022-04-10T01:14:01.470 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2022-04-10T01:14:01.517 INFO:teuthology.orchestra.run.smithi005.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 5.4 MB/s | 323 kB 00:00 2022-04-10T01:14:01.551 INFO:teuthology.orchestra.run.smithi043.stdout:(1/6): jq-1.6-3.el8.x86_64.rpm 2.4 MB/s | 202 kB 00:00 2022-04-10T01:14:01.576 INFO:teuthology.orchestra.run.smithi043.stdout:(2/6): oniguruma-6.8.2-2.el8.x86_64.rpm 7.6 MB/s | 187 kB 00:00 2022-04-10T01:14:01.635 INFO:teuthology.orchestra.run.smithi005.stdout:(4/6): libcephsqlite-17.0.0-11492.g9b3739e7.el8 565 kB/s | 170 kB 00:00 2022-04-10T01:14:01.669 INFO:teuthology.orchestra.run.smithi043.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 3.4 MB/s | 323 kB 00:00 2022-04-10T01:14:01.727 INFO:teuthology.orchestra.run.smithi005.stdout:(5/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 329 kB/s | 69 kB 00:00 2022-04-10T01:14:01.812 INFO:teuthology.orchestra.run.smithi043.stdout:(4/6): libcephsqlite-17.0.0-11492.g9b3739e7.el8 494 kB/s | 170 kB 00:00 2022-04-10T01:14:01.837 INFO:teuthology.orchestra.run.smithi043.stdout:(5/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 411 kB/s | 69 kB 00:00 2022-04-10T01:14:04.386 INFO:teuthology.orchestra.run.smithi043.stdout:(6/6): ceph-test-17.0.0-11492.g9b3739e7.el8.x86 18 MB/s | 52 MB 00:02 2022-04-10T01:14:04.388 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2022-04-10T01:14:04.388 INFO:teuthology.orchestra.run.smithi043.stdout:Total 18 MB/s | 53 MB 00:02 2022-04-10T01:14:04.423 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2022-04-10T01:14:04.445 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2022-04-10T01:14:04.446 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2022-04-10T01:14:04.763 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2022-04-10T01:14:04.766 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2022-04-10T01:14:05.246 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2022-04-10T01:14:05.421 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 1/6 2022-04-10T01:14:05.619 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2022-04-10T01:14:05.765 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2022-04-10T01:14:05.836 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2022-04-10T01:14:06.031 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : jq-1.6-3.el8.x86_64 4/6 2022-04-10T01:14:06.181 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libcephsqlite-2:17.0.0-11492.g9b3739e7.el8.x86_64 5/6 2022-04-10T01:14:07.692 INFO:teuthology.orchestra.run.smithi005.stdout:(6/6): ceph-test-17.0.0-11492.g9b3739e7.el8.x86 8.2 MB/s | 52 MB 00:06 2022-04-10T01:14:07.696 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2022-04-10T01:14:07.696 INFO:teuthology.orchestra.run.smithi005.stdout:Total 8.4 MB/s | 53 MB 00:06 2022-04-10T01:14:07.730 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2022-04-10T01:14:07.753 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2022-04-10T01:14:07.753 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2022-04-10T01:14:08.065 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2022-04-10T01:14:08.069 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2022-04-10T01:14:08.545 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2022-04-10T01:14:08.731 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 1/6 2022-04-10T01:14:08.942 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2022-04-10T01:14:09.088 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2022-04-10T01:14:09.158 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2022-04-10T01:14:09.346 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : jq-1.6-3.el8.x86_64 4/6 2022-04-10T01:14:09.538 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libcephsqlite-2:17.0.0-11492.g9b3739e7.el8.x86_64 5/6 2022-04-10T01:14:11.499 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: libcephsqlite-2:17.0.0-11492.g9b3739e7.el8.x86_64 5/6 2022-04-10T01:14:11.778 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-test-2:17.0.0-11492.g9b3739e7.el8.x86_64 6/6 2022-04-10T01:14:12.034 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-test-2:17.0.0-11492.g9b3739e7.el8.x86_64 6/6 2022-04-10T01:14:12.035 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-test-2:17.0.0-11492.g9b3739e7.el8.x86_64 1/6 2022-04-10T01:14:12.035 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libcephsqlite-2:17.0.0-11492.g9b3739e7.el8.x86_64 2/6 2022-04-10T01:14:12.035 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : jq-1.6-3.el8.x86_64 3/6 2022-04-10T01:14:12.035 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2022-04-10T01:14:12.036 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2022-04-10T01:14:12.469 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 6/6 2022-04-10T01:14:12.469 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:14:12.469 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2022-04-10T01:14:12.469 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-test-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:14:12.470 INFO:teuthology.orchestra.run.smithi043.stdout: jq-1.6-3.el8.x86_64 2022-04-10T01:14:12.470 INFO:teuthology.orchestra.run.smithi043.stdout: libcephsqlite-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:14:12.470 INFO:teuthology.orchestra.run.smithi043.stdout: oniguruma-6.8.2-2.el8.x86_64 2022-04-10T01:14:12.471 INFO:teuthology.orchestra.run.smithi043.stdout: socat-1.7.4.1-1.el8.x86_64 2022-04-10T01:14:12.471 INFO:teuthology.orchestra.run.smithi043.stdout: xmlstarlet-1.6.1-11.el8.x86_64 2022-04-10T01:14:12.471 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:14:12.471 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2022-04-10T01:14:12.671 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph 2022-04-10T01:14:13.260 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:00:58 ago on Sun 10 Apr 2022 01:13:15 AM UTC. 2022-04-10T01:14:14.178 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2022-04-10T01:14:14.182 INFO:teuthology.orchestra.run.smithi043.stdout:======================================================================================== 2022-04-10T01:14:14.182 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repository Size 2022-04-10T01:14:14.182 INFO:teuthology.orchestra.run.smithi043.stdout:======================================================================================== 2022-04-10T01:14:14.182 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2022-04-10T01:14:14.183 INFO:teuthology.orchestra.run.smithi043.stdout: ceph x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 6.5 k 2022-04-10T01:14:14.183 INFO:teuthology.orchestra.run.smithi043.stdout:Installing dependencies: 2022-04-10T01:14:14.183 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mds x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 2.3 M 2022-04-10T01:14:14.183 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 1.5 M 2022-04-10T01:14:14.184 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-modules-core noarch 2:17.0.0-11492.g9b3739e7.el8 ceph-noarch 227 k 2022-04-10T01:14:14.184 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mon x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 3.9 M 2022-04-10T01:14:14.184 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-osd x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 17 M 2022-04-10T01:14:14.184 INFO:teuthology.orchestra.run.smithi043.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2022-04-10T01:14:14.184 INFO:teuthology.orchestra.run.smithi043.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2022-04-10T01:14:14.185 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-Base 237 k 2022-04-10T01:14:14.185 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2022-04-10T01:14:14.185 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2022-04-10T01:14:14.185 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cryptography x86_64 3.2.1-5.el8 CentOS-Base 559 k 2022-04-10T01:14:14.186 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2022-04-10T01:14:14.186 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2022-04-10T01:14:14.186 INFO:teuthology.orchestra.run.smithi043.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2022-04-10T01:14:14.186 INFO:teuthology.orchestra.run.smithi043.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2022-04-10T01:14:14.187 INFO:teuthology.orchestra.run.smithi043.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2022-04-10T01:14:14.187 INFO:teuthology.orchestra.run.smithi043.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2022-04-10T01:14:14.187 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2022-04-10T01:14:14.187 INFO:teuthology.orchestra.run.smithi043.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2022-04-10T01:14:14.187 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 CentOS-AppStream 103 k 2022-04-10T01:14:14.188 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-Base 109 k 2022-04-10T01:14:14.188 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2022-04-10T01:14:14.188 INFO:teuthology.orchestra.run.smithi043.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2022-04-10T01:14:14.188 INFO:teuthology.orchestra.run.smithi043.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2022-04-10T01:14:14.189 INFO:teuthology.orchestra.run.smithi043.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2022-04-10T01:14:14.189 INFO:teuthology.orchestra.run.smithi043.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2022-04-10T01:14:14.189 INFO:teuthology.orchestra.run.smithi043.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2022-04-10T01:14:14.189 INFO:teuthology.orchestra.run.smithi043.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2022-04-10T01:14:14.189 INFO:teuthology.orchestra.run.smithi043.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2022-04-10T01:14:14.190 INFO:teuthology.orchestra.run.smithi043.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2022-04-10T01:14:14.190 INFO:teuthology.orchestra.run.smithi043.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2022-04-10T01:14:14.190 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:14:14.190 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2022-04-10T01:14:14.191 INFO:teuthology.orchestra.run.smithi043.stdout:======================================================================================== 2022-04-10T01:14:14.191 INFO:teuthology.orchestra.run.smithi043.stdout:Install 32 Packages 2022-04-10T01:14:14.191 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:14:14.192 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 28 M 2022-04-10T01:14:14.192 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 113 M 2022-04-10T01:14:14.192 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2022-04-10T01:14:14.389 INFO:teuthology.orchestra.run.smithi043.stdout:(1/32): ceph-17.0.0-11492.g9b3739e7.el8.x86_64. 32 kB/s | 6.5 kB 00:00 2022-04-10T01:14:14.597 INFO:teuthology.orchestra.run.smithi043.stdout:(2/32): ceph-mgr-17.0.0-11492.g9b3739e7.el8.x86 3.8 MB/s | 1.5 MB 00:00 2022-04-10T01:14:14.664 INFO:teuthology.orchestra.run.smithi043.stdout:(3/32): ceph-mds-17.0.0-11492.g9b3739e7.el8.x86 4.8 MB/s | 2.3 MB 00:00 2022-04-10T01:14:14.731 INFO:teuthology.orchestra.run.smithi043.stdout:(4/32): ceph-mgr-modules-core-17.0.0-11492.g9b3 3.3 MB/s | 227 kB 00:00 2022-04-10T01:14:14.789 INFO:teuthology.orchestra.run.smithi043.stdout:(5/32): python3-mako-1.0.6-13.el8.noarch.rpm 2.6 MB/s | 157 kB 00:00 2022-04-10T01:14:14.815 INFO:teuthology.orchestra.run.smithi043.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 1.5 MB/s | 39 kB 00:00 2022-04-10T01:14:14.840 INFO:teuthology.orchestra.run.smithi043.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 4.0 MB/s | 103 kB 00:00 2022-04-10T01:14:14.866 INFO:teuthology.orchestra.run.smithi043.stdout:(8/32): python3-pytz-2017.2-9.el8.noarch.rpm 2.1 MB/s | 54 kB 00:00 2022-04-10T01:14:14.924 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: libcephsqlite-2:17.0.0-11492.g9b3739e7.el8.x86_64 5/6 2022-04-10T01:14:14.932 INFO:teuthology.orchestra.run.smithi043.stdout:(9/32): ceph-mon-17.0.0-11492.g9b3739e7.el8.x86 7.1 MB/s | 3.9 MB 00:00 2022-04-10T01:14:14.966 INFO:teuthology.orchestra.run.smithi043.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 4.5 MB/s | 457 kB 00:00 2022-04-10T01:14:14.992 INFO:teuthology.orchestra.run.smithi043.stdout:(11/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 4.0 MB/s | 237 kB 00:00 2022-04-10T01:14:15.017 INFO:teuthology.orchestra.run.smithi043.stdout:(12/32): python3-cryptography-3.2.1-5.el8.x86_6 11 MB/s | 559 kB 00:00 2022-04-10T01:14:15.042 INFO:teuthology.orchestra.run.smithi043.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 2.1 MB/s | 109 kB 00:00 2022-04-10T01:14:15.151 INFO:teuthology.orchestra.run.smithi043.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 326 kB/s | 44 kB 00:00 2022-04-10T01:14:15.152 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-test-2:17.0.0-11492.g9b3739e7.el8.x86_64 6/6 2022-04-10T01:14:15.209 INFO:teuthology.orchestra.run.smithi043.stdout:(15/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.1 MB/s | 185 kB 00:00 2022-04-10T01:14:15.393 INFO:teuthology.orchestra.run.smithi043.stdout:(16/32): python3-cherrypy-18.4.0-1.el8.noarch.r 2.1 MB/s | 384 kB 00:00 2022-04-10T01:14:15.418 INFO:teuthology.orchestra.run.smithi043.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 646 kB/s | 173 kB 00:00 2022-04-10T01:14:15.423 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-test-2:17.0.0-11492.g9b3739e7.el8.x86_64 6/6 2022-04-10T01:14:15.423 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-test-2:17.0.0-11492.g9b3739e7.el8.x86_64 1/6 2022-04-10T01:14:15.423 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libcephsqlite-2:17.0.0-11492.g9b3739e7.el8.x86_64 2/6 2022-04-10T01:14:15.425 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : jq-1.6-3.el8.x86_64 3/6 2022-04-10T01:14:15.425 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2022-04-10T01:14:15.425 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2022-04-10T01:14:15.710 INFO:teuthology.orchestra.run.smithi043.stdout:(18/32): ceph-osd-17.0.0-11492.g9b3739e7.el8.x8 15 MB/s | 17 MB 00:01 2022-04-10T01:14:15.727 INFO:teuthology.orchestra.run.smithi043.stdout:(19/32): python3-jaraco-6.2-6.el8.noarch.rpm 33 kB/s | 11 kB 00:00 2022-04-10T01:14:15.744 INFO:teuthology.orchestra.run.smithi043.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 54 kB/s | 18 kB 00:00 2022-04-10T01:14:15.794 INFO:teuthology.orchestra.run.smithi043.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 582 kB/s | 49 kB 00:00 2022-04-10T01:14:15.845 INFO:teuthology.orchestra.run.smithi043.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 501 kB/s | 59 kB 00:00 2022-04-10T01:14:15.920 INFO:teuthology.orchestra.run.smithi043.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 124 kB/s | 16 kB 00:00 2022-04-10T01:14:16.010 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 6/6 2022-04-10T01:14:16.011 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:14:16.011 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2022-04-10T01:14:16.011 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-test-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:14:16.011 INFO:teuthology.orchestra.run.smithi005.stdout: jq-1.6-3.el8.x86_64 2022-04-10T01:14:16.011 INFO:teuthology.orchestra.run.smithi005.stdout: libcephsqlite-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:14:16.012 INFO:teuthology.orchestra.run.smithi005.stdout: oniguruma-6.8.2-2.el8.x86_64 2022-04-10T01:14:16.012 INFO:teuthology.orchestra.run.smithi005.stdout: socat-1.7.4.1-1.el8.x86_64 2022-04-10T01:14:16.012 INFO:teuthology.orchestra.run.smithi005.stdout: xmlstarlet-1.6.1-11.el8.x86_64 2022-04-10T01:14:16.012 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:14:16.013 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2022-04-10T01:14:16.037 INFO:teuthology.orchestra.run.smithi043.stdout:(24/32): python3-pecan-1.3.2-9.el8.noarch.rpm 967 kB/s | 283 kB 00:00 2022-04-10T01:14:16.243 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph 2022-04-10T01:14:16.279 INFO:teuthology.orchestra.run.smithi043.stdout:(25/32): python3-tempora-1.14.1-5.el8.noarch.rp 121 kB/s | 29 kB 00:00 2022-04-10T01:14:16.296 INFO:teuthology.orchestra.run.smithi043.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 64 kB/s | 24 kB 00:00 2022-04-10T01:14:16.321 INFO:teuthology.orchestra.run.smithi043.stdout:(27/32): python3-simplegeneric-0.8.1-17.el8.noa 39 kB/s | 19 kB 00:00 2022-04-10T01:14:16.381 INFO:teuthology.orchestra.run.smithi043.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 273 kB/s | 27 kB 00:00 2022-04-10T01:14:16.680 INFO:teuthology.orchestra.run.smithi043.stdout:(29/32): python3-webtest-2.0.33-1.el8.noarch.rp 284 kB/s | 85 kB 00:00 2022-04-10T01:14:16.713 INFO:teuthology.orchestra.run.smithi043.stdout:(30/32): python3-waitress-1.2.1-2.el8.1.noarch. 577 kB/s | 241 kB 00:00 2022-04-10T01:14:16.739 INFO:teuthology.orchestra.run.smithi043.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 602 kB/s | 251 kB 00:00 2022-04-10T01:14:16.789 INFO:teuthology.orchestra.run.smithi043.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 211 kB/s | 23 kB 00:00 2022-04-10T01:14:16.793 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2022-04-10T01:14:16.793 INFO:teuthology.orchestra.run.smithi043.stdout:Total 11 MB/s | 28 MB 00:02 2022-04-10T01:14:16.829 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:01 ago on Sun 10 Apr 2022 01:13:15 AM UTC. 2022-04-10T01:14:16.879 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2022-04-10T01:14:16.919 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2022-04-10T01:14:16.919 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2022-04-10T01:14:17.257 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2022-04-10T01:14:17.260 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2022-04-10T01:14:17.737 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2022-04-10T01:14:17.741 INFO:teuthology.orchestra.run.smithi005.stdout:======================================================================================== 2022-04-10T01:14:17.741 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repository Size 2022-04-10T01:14:17.741 INFO:teuthology.orchestra.run.smithi005.stdout:======================================================================================== 2022-04-10T01:14:17.741 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2022-04-10T01:14:17.742 INFO:teuthology.orchestra.run.smithi005.stdout: ceph x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 6.5 k 2022-04-10T01:14:17.742 INFO:teuthology.orchestra.run.smithi005.stdout:Installing dependencies: 2022-04-10T01:14:17.742 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mds x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 2.3 M 2022-04-10T01:14:17.743 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 1.5 M 2022-04-10T01:14:17.743 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-modules-core noarch 2:17.0.0-11492.g9b3739e7.el8 ceph-noarch 227 k 2022-04-10T01:14:17.743 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mon x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 3.9 M 2022-04-10T01:14:17.743 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-osd x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 17 M 2022-04-10T01:14:17.744 INFO:teuthology.orchestra.run.smithi005.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2022-04-10T01:14:17.744 INFO:teuthology.orchestra.run.smithi005.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2022-04-10T01:14:17.744 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-Base 237 k 2022-04-10T01:14:17.744 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2022-04-10T01:14:17.744 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2022-04-10T01:14:17.745 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cryptography x86_64 3.2.1-5.el8 CentOS-Base 559 k 2022-04-10T01:14:17.745 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2022-04-10T01:14:17.745 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2022-04-10T01:14:17.745 INFO:teuthology.orchestra.run.smithi005.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2022-04-10T01:14:17.746 INFO:teuthology.orchestra.run.smithi005.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2022-04-10T01:14:17.746 INFO:teuthology.orchestra.run.smithi005.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2022-04-10T01:14:17.746 INFO:teuthology.orchestra.run.smithi005.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2022-04-10T01:14:17.746 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2022-04-10T01:14:17.746 INFO:teuthology.orchestra.run.smithi005.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2022-04-10T01:14:17.747 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 CentOS-AppStream 103 k 2022-04-10T01:14:17.747 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-Base 109 k 2022-04-10T01:14:17.747 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2022-04-10T01:14:17.747 INFO:teuthology.orchestra.run.smithi005.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2022-04-10T01:14:17.748 INFO:teuthology.orchestra.run.smithi005.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2022-04-10T01:14:17.748 INFO:teuthology.orchestra.run.smithi005.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2022-04-10T01:14:17.748 INFO:teuthology.orchestra.run.smithi005.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2022-04-10T01:14:17.748 INFO:teuthology.orchestra.run.smithi005.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2022-04-10T01:14:17.749 INFO:teuthology.orchestra.run.smithi005.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2022-04-10T01:14:17.749 INFO:teuthology.orchestra.run.smithi005.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2022-04-10T01:14:17.749 INFO:teuthology.orchestra.run.smithi005.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2022-04-10T01:14:17.749 INFO:teuthology.orchestra.run.smithi005.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2022-04-10T01:14:17.750 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:14:17.750 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2022-04-10T01:14:17.750 INFO:teuthology.orchestra.run.smithi005.stdout:======================================================================================== 2022-04-10T01:14:17.750 INFO:teuthology.orchestra.run.smithi005.stdout:Install 32 Packages 2022-04-10T01:14:17.751 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:14:17.752 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 28 M 2022-04-10T01:14:17.752 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 113 M 2022-04-10T01:14:17.752 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2022-04-10T01:14:17.877 INFO:teuthology.orchestra.run.smithi005.stdout:(1/32): ceph-17.0.0-11492.g9b3739e7.el8.x86_64. 50 kB/s | 6.5 kB 00:00 2022-04-10T01:14:18.119 INFO:teuthology.orchestra.run.smithi005.stdout:(2/32): ceph-mds-17.0.0-11492.g9b3739e7.el8.x86 6.2 MB/s | 2.3 MB 00:00 2022-04-10T01:14:18.152 INFO:teuthology.orchestra.run.smithi005.stdout:(3/32): ceph-mgr-17.0.0-11492.g9b3739e7.el8.x86 3.8 MB/s | 1.5 MB 00:00 2022-04-10T01:14:18.228 INFO:teuthology.orchestra.run.smithi005.stdout:(4/32): ceph-mgr-modules-core-17.0.0-11492.g9b3 3.0 MB/s | 227 kB 00:00 2022-04-10T01:14:18.286 INFO:teuthology.orchestra.run.smithi005.stdout:(5/32): python3-mako-1.0.6-13.el8.noarch.rpm 2.6 MB/s | 157 kB 00:00 2022-04-10T01:14:18.303 INFO:teuthology.orchestra.run.smithi005.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 2.3 MB/s | 39 kB 00:00 2022-04-10T01:14:18.328 INFO:teuthology.orchestra.run.smithi005.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 4.0 MB/s | 103 kB 00:00 2022-04-10T01:14:18.372 INFO:teuthology.orchestra.run.smithi005.stdout:(8/32): python3-pytz-2017.2-9.el8.noarch.rpm 1.2 MB/s | 54 kB 00:00 2022-04-10T01:14:18.404 INFO:teuthology.orchestra.run.smithi005.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 14 MB/s | 457 kB 00:00 2022-04-10T01:14:18.421 INFO:teuthology.orchestra.run.smithi005.stdout:(10/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 14 MB/s | 237 kB 00:00 2022-04-10T01:14:18.455 INFO:teuthology.orchestra.run.smithi005.stdout:(11/32): python3-cryptography-3.2.1-5.el8.x86_6 16 MB/s | 559 kB 00:00 2022-04-10T01:14:18.455 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2022-04-10T01:14:18.480 INFO:teuthology.orchestra.run.smithi005.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 4.3 MB/s | 109 kB 00:00 2022-04-10T01:14:18.705 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2022-04-10T01:14:18.739 INFO:teuthology.orchestra.run.smithi005.stdout:(13/32): ceph-mon-17.0.0-11492.g9b3739e7.el8.x8 4.5 MB/s | 3.9 MB 00:00 2022-04-10T01:14:18.755 INFO:teuthology.orchestra.run.smithi005.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 158 kB/s | 44 kB 00:00 2022-04-10T01:14:18.938 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2022-04-10T01:14:19.048 INFO:teuthology.orchestra.run.smithi005.stdout:(15/32): ceph-osd-17.0.0-11492.g9b3739e7.el8.x8 18 MB/s | 17 MB 00:00 2022-04-10T01:14:19.073 INFO:teuthology.orchestra.run.smithi005.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 555 kB/s | 185 kB 00:00 2022-04-10T01:14:19.098 INFO:teuthology.orchestra.run.smithi005.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 504 kB/s | 173 kB 00:00 2022-04-10T01:14:19.123 INFO:teuthology.orchestra.run.smithi005.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 220 kB/s | 11 kB 00:00 2022-04-10T01:14:19.149 INFO:teuthology.orchestra.run.smithi005.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 348 kB/s | 18 kB 00:00 2022-04-10T01:14:19.190 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2022-04-10T01:14:19.207 INFO:teuthology.orchestra.run.smithi005.stdout:(20/32): python3-more-itertools-7.2.0-3.el8.noa 1.0 MB/s | 59 kB 00:00 2022-04-10T01:14:19.233 INFO:teuthology.orchestra.run.smithi005.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 446 kB/s | 49 kB 00:00 2022-04-10T01:14:19.283 INFO:teuthology.orchestra.run.smithi005.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 312 kB/s | 16 kB 00:00 2022-04-10T01:14:19.339 INFO:teuthology.orchestra.run.smithi005.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 2.1 MB/s | 283 kB 00:00 2022-04-10T01:14:19.367 INFO:teuthology.orchestra.run.smithi005.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 223 kB/s | 19 kB 00:00 2022-04-10T01:14:19.392 INFO:teuthology.orchestra.run.smithi005.stdout:(25/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.1 MB/s | 384 kB 00:00 2022-04-10T01:14:19.418 INFO:teuthology.orchestra.run.smithi005.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 305 kB/s | 24 kB 00:00 2022-04-10T01:14:19.428 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 4/32 2022-04-10T01:14:19.443 INFO:teuthology.orchestra.run.smithi005.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 388 kB/s | 29 kB 00:00 2022-04-10T01:14:19.468 INFO:teuthology.orchestra.run.smithi005.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 365 kB/s | 27 kB 00:00 2022-04-10T01:14:19.485 INFO:teuthology.orchestra.run.smithi005.stdout:(29/32): python3-waitress-1.2.1-2.el8.1.noarch. 3.5 MB/s | 241 kB 00:00 2022-04-10T01:14:19.519 INFO:teuthology.orchestra.run.smithi005.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.7 MB/s | 85 kB 00:00 2022-04-10T01:14:19.544 INFO:teuthology.orchestra.run.smithi005.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 392 kB/s | 23 kB 00:00 2022-04-10T01:14:19.626 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2022-04-10T01:14:19.650 INFO:teuthology.orchestra.run.smithi005.stdout:(32/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.2 MB/s | 251 kB 00:00 2022-04-10T01:14:19.654 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2022-04-10T01:14:19.654 INFO:teuthology.orchestra.run.smithi005.stdout:Total 15 MB/s | 28 MB 00:01 2022-04-10T01:14:19.740 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2022-04-10T01:14:19.779 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2022-04-10T01:14:19.779 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2022-04-10T01:14:19.809 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2022-04-10T01:14:20.033 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2022-04-10T01:14:20.092 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2022-04-10T01:14:20.095 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2022-04-10T01:14:20.194 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2022-04-10T01:14:20.415 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2022-04-10T01:14:20.660 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2022-04-10T01:14:20.889 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2022-04-10T01:14:21.106 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2022-04-10T01:14:21.336 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2022-04-10T01:14:21.512 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 13/32 2022-04-10T01:14:21.630 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2022-04-10T01:14:21.775 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 14/32 2022-04-10T01:14:21.920 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2022-04-10T01:14:21.979 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2022-04-10T01:14:22.148 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2022-04-10T01:14:22.209 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2022-04-10T01:14:22.369 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 4/32 2022-04-10T01:14:22.456 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2022-04-10T01:14:22.566 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2022-04-10T01:14:22.733 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2022-04-10T01:14:22.776 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2022-04-10T01:14:22.965 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2022-04-10T01:14:23.028 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2022-04-10T01:14:23.143 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2022-04-10T01:14:23.183 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 20/32 2022-04-10T01:14:23.354 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2022-04-10T01:14:23.436 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2022-04-10T01:14:23.624 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2022-04-10T01:14:23.698 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2022-04-10T01:14:23.879 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2022-04-10T01:14:23.951 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2022-04-10T01:14:24.129 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2022-04-10T01:14:24.191 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2022-04-10T01:14:24.510 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 13/32 2022-04-10T01:14:24.605 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 25/32 2022-04-10T01:14:24.791 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 14/32 2022-04-10T01:14:24.945 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2022-04-10T01:14:24.983 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2022-04-10T01:14:25.226 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2022-04-10T01:14:25.340 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-11492.g9b3739e7.el8 27/32 2022-04-10T01:14:25.493 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-2:17.0.0-11492.g9b3739e7.el8.x86_64 28/32 2022-04-10T01:14:25.514 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2022-04-10T01:14:25.882 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2022-04-10T01:14:26.096 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2022-04-10T01:14:26.282 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 20/32 2022-04-10T01:14:26.484 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2022-04-10T01:14:26.771 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2022-04-10T01:14:27.025 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2022-04-10T01:14:27.113 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mgr-2:17.0.0-11492.g9b3739e7.el8.x86_64 28/32 2022-04-10T01:14:27.272 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2022-04-10T01:14:27.309 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-osd-2:17.0.0-11492.g9b3739e7.el8.x86_64 29/32 2022-04-10T01:14:27.686 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 25/32 2022-04-10T01:14:27.721 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-osd-2:17.0.0-11492.g9b3739e7.el8.x86_64 29/32 2022-04-10T01:14:27.943 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mon-2:17.0.0-11492.g9b3739e7.el8.x86_64 30/32 2022-04-10T01:14:28.041 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2022-04-10T01:14:28.174 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mon-2:17.0.0-11492.g9b3739e7.el8.x86_64 30/32 2022-04-10T01:14:28.362 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mds-2:17.0.0-11492.g9b3739e7.el8.x86_64 31/32 2022-04-10T01:14:28.379 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mds-2:17.0.0-11492.g9b3739e7.el8.x86_64 31/32 2022-04-10T01:14:28.490 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-11492.g9b3739e7.el8 27/32 2022-04-10T01:14:28.502 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-2:17.0.0-11492.g9b3739e7.el8.x86_64 32/32 2022-04-10T01:14:28.677 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mgr-2:17.0.0-11492.g9b3739e7.el8.x86_64 28/32 2022-04-10T01:14:28.900 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-2:17.0.0-11492.g9b3739e7.el8.x86_64 32/32 2022-04-10T01:14:28.901 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-2:17.0.0-11492.g9b3739e7.el8.x86_64 1/32 2022-04-10T01:14:28.901 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mds-2:17.0.0-11492.g9b3739e7.el8.x86_64 2/32 2022-04-10T01:14:28.901 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-2:17.0.0-11492.g9b3739e7.el8.x86_64 3/32 2022-04-10T01:14:28.902 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mon-2:17.0.0-11492.g9b3739e7.el8.x86_64 4/32 2022-04-10T01:14:28.902 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-osd-2:17.0.0-11492.g9b3739e7.el8.x86_64 5/32 2022-04-10T01:14:28.902 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-11492.g9b3739e7.el8 6/32 2022-04-10T01:14:28.902 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 7/32 2022-04-10T01:14:28.902 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2022-04-10T01:14:28.903 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2022-04-10T01:14:28.903 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 10/32 2022-04-10T01:14:28.903 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2022-04-10T01:14:28.903 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 12/32 2022-04-10T01:14:28.904 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 13/32 2022-04-10T01:14:28.904 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2022-04-10T01:14:28.904 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2022-04-10T01:14:28.904 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2022-04-10T01:14:28.904 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2022-04-10T01:14:28.905 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2022-04-10T01:14:28.905 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2022-04-10T01:14:28.905 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2022-04-10T01:14:28.905 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2022-04-10T01:14:28.906 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2022-04-10T01:14:28.906 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2022-04-10T01:14:28.906 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2022-04-10T01:14:28.906 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2022-04-10T01:14:28.906 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2022-04-10T01:14:28.907 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2022-04-10T01:14:28.907 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2022-04-10T01:14:28.908 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 29/32 2022-04-10T01:14:28.908 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2022-04-10T01:14:28.908 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2022-04-10T01:14:29.700 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2022-04-10T01:14:29.700 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:14:29.700 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2022-04-10T01:14:29.701 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:14:29.701 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mds-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:14:29.701 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:14:29.701 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-modules-core-2:17.0.0-11492.g9b3739e7.el8.noarch 2022-04-10T01:14:29.702 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mon-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:14:29.702 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-osd-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:14:29.702 INFO:teuthology.orchestra.run.smithi043.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2022-04-10T01:14:29.702 INFO:teuthology.orchestra.run.smithi043.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2022-04-10T01:14:29.702 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cffi-1.11.5-5.el8.x86_64 2022-04-10T01:14:29.703 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cheroot-8.5.2-1.el8.noarch 2022-04-10T01:14:29.703 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2022-04-10T01:14:29.703 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2022-04-10T01:14:29.703 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-6.2-6.el8.noarch 2022-04-10T01:14:29.704 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2022-04-10T01:14:29.704 INFO:teuthology.orchestra.run.smithi043.stdout: python3-logutils-0.3.5-11.el8.noarch 2022-04-10T01:14:29.704 INFO:teuthology.orchestra.run.smithi043.stdout: python3-mako-1.0.6-13.el8.noarch 2022-04-10T01:14:29.704 INFO:teuthology.orchestra.run.smithi043.stdout: python3-markupsafe-0.23-19.el8.x86_64 2022-04-10T01:14:29.704 INFO:teuthology.orchestra.run.smithi043.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2022-04-10T01:14:29.705 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pecan-1.3.2-9.el8.noarch 2022-04-10T01:14:29.705 INFO:teuthology.orchestra.run.smithi043.stdout: python3-portend-2.6-1.el8.noarch 2022-04-10T01:14:29.705 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2022-04-10T01:14:29.705 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pycparser-2.14-14.el8.noarch 2022-04-10T01:14:29.706 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pytz-2017.2-9.el8.noarch 2022-04-10T01:14:29.706 INFO:teuthology.orchestra.run.smithi043.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2022-04-10T01:14:29.706 INFO:teuthology.orchestra.run.smithi043.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2022-04-10T01:14:29.706 INFO:teuthology.orchestra.run.smithi043.stdout: python3-tempora-1.14.1-5.el8.noarch 2022-04-10T01:14:29.707 INFO:teuthology.orchestra.run.smithi043.stdout: python3-trustme-0.6.0-4.el8.noarch 2022-04-10T01:14:29.707 INFO:teuthology.orchestra.run.smithi043.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2022-04-10T01:14:29.707 INFO:teuthology.orchestra.run.smithi043.stdout: python3-webob-1.8.5-1.el8.1.noarch 2022-04-10T01:14:29.707 INFO:teuthology.orchestra.run.smithi043.stdout: python3-webtest-2.0.33-1.el8.noarch 2022-04-10T01:14:29.707 INFO:teuthology.orchestra.run.smithi043.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2022-04-10T01:14:29.708 INFO:teuthology.orchestra.run.smithi043.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2022-04-10T01:14:29.708 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:14:29.708 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2022-04-10T01:14:29.896 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph-base 2022-04-10T01:14:30.304 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-mgr-2:17.0.0-11492.g9b3739e7.el8.x86_64 28/32 2022-04-10T01:14:30.499 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:15 ago on Sun 10 Apr 2022 01:13:15 AM UTC. 2022-04-10T01:14:30.518 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-osd-2:17.0.0-11492.g9b3739e7.el8.x86_64 29/32 2022-04-10T01:14:31.016 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-osd-2:17.0.0-11492.g9b3739e7.el8.x86_64 29/32 2022-04-10T01:14:31.329 INFO:teuthology.orchestra.run.smithi043.stdout:Package ceph-base-2:17.0.0-11492.g9b3739e7.el8.x86_64 is already installed. 2022-04-10T01:14:31.387 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2022-04-10T01:14:31.388 INFO:teuthology.orchestra.run.smithi043.stdout:Nothing to do. 2022-04-10T01:14:31.389 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2022-04-10T01:14:31.540 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mon-2:17.0.0-11492.g9b3739e7.el8.x86_64 30/32 2022-04-10T01:14:31.724 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install cephadm 2022-04-10T01:14:31.777 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-mon-2:17.0.0-11492.g9b3739e7.el8.x86_64 30/32 2022-04-10T01:14:31.960 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mds-2:17.0.0-11492.g9b3739e7.el8.x86_64 31/32 2022-04-10T01:14:31.979 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-mds-2:17.0.0-11492.g9b3739e7.el8.x86_64 31/32 2022-04-10T01:14:32.134 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-2:17.0.0-11492.g9b3739e7.el8.x86_64 32/32 2022-04-10T01:14:32.315 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:17 ago on Sun 10 Apr 2022 01:13:15 AM UTC. 2022-04-10T01:14:32.502 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-2:17.0.0-11492.g9b3739e7.el8.x86_64 32/32 2022-04-10T01:14:32.502 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-2:17.0.0-11492.g9b3739e7.el8.x86_64 1/32 2022-04-10T01:14:32.503 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mds-2:17.0.0-11492.g9b3739e7.el8.x86_64 2/32 2022-04-10T01:14:32.503 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mgr-2:17.0.0-11492.g9b3739e7.el8.x86_64 3/32 2022-04-10T01:14:32.503 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mon-2:17.0.0-11492.g9b3739e7.el8.x86_64 4/32 2022-04-10T01:14:32.504 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-osd-2:17.0.0-11492.g9b3739e7.el8.x86_64 5/32 2022-04-10T01:14:32.504 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-11492.g9b3739e7.el8 6/32 2022-04-10T01:14:32.504 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 7/32 2022-04-10T01:14:32.504 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2022-04-10T01:14:32.504 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2022-04-10T01:14:32.505 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 10/32 2022-04-10T01:14:32.505 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2022-04-10T01:14:32.505 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 12/32 2022-04-10T01:14:32.505 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 13/32 2022-04-10T01:14:32.506 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2022-04-10T01:14:32.506 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2022-04-10T01:14:32.506 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2022-04-10T01:14:32.507 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2022-04-10T01:14:32.507 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2022-04-10T01:14:32.507 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2022-04-10T01:14:32.507 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2022-04-10T01:14:32.508 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2022-04-10T01:14:32.508 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2022-04-10T01:14:32.508 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2022-04-10T01:14:32.508 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2022-04-10T01:14:32.508 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2022-04-10T01:14:32.509 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2022-04-10T01:14:32.509 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2022-04-10T01:14:32.509 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2022-04-10T01:14:32.509 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 29/32 2022-04-10T01:14:32.510 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2022-04-10T01:14:32.510 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2022-04-10T01:14:33.205 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2022-04-10T01:14:33.206 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2022-04-10T01:14:33.206 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repository Size 2022-04-10T01:14:33.206 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2022-04-10T01:14:33.207 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2022-04-10T01:14:33.207 INFO:teuthology.orchestra.run.smithi043.stdout: cephadm noarch 2:17.0.0-11492.g9b3739e7.el8 ceph-noarch 72 k 2022-04-10T01:14:33.207 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:14:33.207 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2022-04-10T01:14:33.207 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2022-04-10T01:14:33.208 INFO:teuthology.orchestra.run.smithi043.stdout:Install 1 Package 2022-04-10T01:14:33.208 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:14:33.208 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 72 k 2022-04-10T01:14:33.208 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 324 k 2022-04-10T01:14:33.209 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2022-04-10T01:14:33.388 INFO:teuthology.orchestra.run.smithi043.stdout:cephadm-17.0.0-11492.g9b3739e7.el8.noarch.rpm 395 kB/s | 72 kB 00:00 2022-04-10T01:14:33.389 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2022-04-10T01:14:33.389 INFO:teuthology.orchestra.run.smithi043.stdout:Total 392 kB/s | 72 kB 00:00 2022-04-10T01:14:33.389 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2022-04-10T01:14:33.397 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2022-04-10T01:14:33.398 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2022-04-10T01:14:33.399 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2022-04-10T01:14:33.402 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2022-04-10T01:14:33.429 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2022-04-10T01:14:33.429 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:14:33.429 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2022-04-10T01:14:33.429 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:14:33.430 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mds-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:14:33.430 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:14:33.430 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-modules-core-2:17.0.0-11492.g9b3739e7.el8.noarch 2022-04-10T01:14:33.431 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mon-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:14:33.431 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-osd-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:14:33.431 INFO:teuthology.orchestra.run.smithi005.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2022-04-10T01:14:33.431 INFO:teuthology.orchestra.run.smithi005.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2022-04-10T01:14:33.431 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cffi-1.11.5-5.el8.x86_64 2022-04-10T01:14:33.432 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cheroot-8.5.2-1.el8.noarch 2022-04-10T01:14:33.432 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2022-04-10T01:14:33.432 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2022-04-10T01:14:33.432 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco-6.2-6.el8.noarch 2022-04-10T01:14:33.433 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2022-04-10T01:14:33.433 INFO:teuthology.orchestra.run.smithi005.stdout: python3-logutils-0.3.5-11.el8.noarch 2022-04-10T01:14:33.433 INFO:teuthology.orchestra.run.smithi005.stdout: python3-mako-1.0.6-13.el8.noarch 2022-04-10T01:14:33.433 INFO:teuthology.orchestra.run.smithi005.stdout: python3-markupsafe-0.23-19.el8.x86_64 2022-04-10T01:14:33.433 INFO:teuthology.orchestra.run.smithi005.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2022-04-10T01:14:33.434 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pecan-1.3.2-9.el8.noarch 2022-04-10T01:14:33.434 INFO:teuthology.orchestra.run.smithi005.stdout: python3-portend-2.6-1.el8.noarch 2022-04-10T01:14:33.434 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2022-04-10T01:14:33.434 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pycparser-2.14-14.el8.noarch 2022-04-10T01:14:33.435 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pytz-2017.2-9.el8.noarch 2022-04-10T01:14:33.435 INFO:teuthology.orchestra.run.smithi005.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2022-04-10T01:14:33.435 INFO:teuthology.orchestra.run.smithi005.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2022-04-10T01:14:33.435 INFO:teuthology.orchestra.run.smithi005.stdout: python3-tempora-1.14.1-5.el8.noarch 2022-04-10T01:14:33.435 INFO:teuthology.orchestra.run.smithi005.stdout: python3-trustme-0.6.0-4.el8.noarch 2022-04-10T01:14:33.436 INFO:teuthology.orchestra.run.smithi005.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2022-04-10T01:14:33.436 INFO:teuthology.orchestra.run.smithi005.stdout: python3-webob-1.8.5-1.el8.1.noarch 2022-04-10T01:14:33.436 INFO:teuthology.orchestra.run.smithi005.stdout: python3-webtest-2.0.33-1.el8.noarch 2022-04-10T01:14:33.436 INFO:teuthology.orchestra.run.smithi005.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2022-04-10T01:14:33.437 INFO:teuthology.orchestra.run.smithi005.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2022-04-10T01:14:33.437 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:14:33.437 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2022-04-10T01:14:33.461 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2022-04-10T01:14:33.683 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-base 2022-04-10T01:14:34.279 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:19 ago on Sun 10 Apr 2022 01:13:15 AM UTC. 2022-04-10T01:14:35.091 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: cephadm-2:17.0.0-11492.g9b3739e7.el8.noarch 1/1 2022-04-10T01:14:35.129 INFO:teuthology.orchestra.run.smithi005.stdout:Package ceph-base-2:17.0.0-11492.g9b3739e7.el8.x86_64 is already installed. 2022-04-10T01:14:35.185 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2022-04-10T01:14:35.186 INFO:teuthology.orchestra.run.smithi005.stdout:Nothing to do. 2022-04-10T01:14:35.186 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2022-04-10T01:14:35.262 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : cephadm-2:17.0.0-11492.g9b3739e7.el8.noarch 1/1 2022-04-10T01:14:35.265 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install cephadm 2022-04-10T01:14:35.453 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: cephadm-2:17.0.0-11492.g9b3739e7.el8.noarch 1/1 2022-04-10T01:14:35.856 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:20 ago on Sun 10 Apr 2022 01:13:15 AM UTC. 2022-04-10T01:14:35.864 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : cephadm-2:17.0.0-11492.g9b3739e7.el8.noarch 1/1 2022-04-10T01:14:35.864 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:14:35.864 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2022-04-10T01:14:35.864 INFO:teuthology.orchestra.run.smithi043.stdout: cephadm-2:17.0.0-11492.g9b3739e7.el8.noarch 2022-04-10T01:14:35.865 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:14:35.865 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2022-04-10T01:14:36.101 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph-immutable-object-cache 2022-04-10T01:14:36.746 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2022-04-10T01:14:36.747 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2022-04-10T01:14:36.747 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repository Size 2022-04-10T01:14:36.748 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2022-04-10T01:14:36.748 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2022-04-10T01:14:36.748 INFO:teuthology.orchestra.run.smithi005.stdout: cephadm noarch 2:17.0.0-11492.g9b3739e7.el8 ceph-noarch 72 k 2022-04-10T01:14:36.748 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:14:36.749 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2022-04-10T01:14:36.749 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2022-04-10T01:14:36.749 INFO:teuthology.orchestra.run.smithi005.stdout:Install 1 Package 2022-04-10T01:14:36.749 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:14:36.750 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 72 k 2022-04-10T01:14:36.750 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 324 k 2022-04-10T01:14:36.750 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2022-04-10T01:14:36.753 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:21 ago on Sun 10 Apr 2022 01:13:15 AM UTC. 2022-04-10T01:14:36.926 INFO:teuthology.orchestra.run.smithi005.stdout:cephadm-17.0.0-11492.g9b3739e7.el8.noarch.rpm 402 kB/s | 72 kB 00:00 2022-04-10T01:14:36.927 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2022-04-10T01:14:36.928 INFO:teuthology.orchestra.run.smithi005.stdout:Total 398 kB/s | 72 kB 00:00 2022-04-10T01:14:36.928 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2022-04-10T01:14:36.936 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2022-04-10T01:14:36.936 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2022-04-10T01:14:36.938 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2022-04-10T01:14:36.941 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2022-04-10T01:14:36.985 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2022-04-10T01:14:37.668 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2022-04-10T01:14:37.670 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2022-04-10T01:14:37.670 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repo Size 2022-04-10T01:14:37.670 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2022-04-10T01:14:37.670 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2022-04-10T01:14:37.671 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 169 k 2022-04-10T01:14:37.671 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:14:37.671 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2022-04-10T01:14:37.671 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2022-04-10T01:14:37.672 INFO:teuthology.orchestra.run.smithi043.stdout:Install 1 Package 2022-04-10T01:14:37.672 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:14:37.672 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 169 k 2022-04-10T01:14:37.672 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 567 k 2022-04-10T01:14:37.673 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2022-04-10T01:14:38.060 INFO:teuthology.orchestra.run.smithi043.stdout:ceph-immutable-object-cache-17.0.0-11492.g9b373 433 kB/s | 169 kB 00:00 2022-04-10T01:14:38.061 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2022-04-10T01:14:38.062 INFO:teuthology.orchestra.run.smithi043.stdout:Total 432 kB/s | 169 kB 00:00 2022-04-10T01:14:38.062 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2022-04-10T01:14:38.074 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2022-04-10T01:14:38.074 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2022-04-10T01:14:38.127 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2022-04-10T01:14:38.130 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2022-04-10T01:14:38.250 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2022-04-10T01:14:38.443 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-11492.g9b3739e7 1/1 2022-04-10T01:14:38.585 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: cephadm-2:17.0.0-11492.g9b3739e7.el8.noarch 1/1 2022-04-10T01:14:38.779 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : cephadm-2:17.0.0-11492.g9b3739e7.el8.noarch 1/1 2022-04-10T01:14:38.815 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-11492.g9b3739e7 1/1 2022-04-10T01:14:38.968 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: cephadm-2:17.0.0-11492.g9b3739e7.el8.noarch 1/1 2022-04-10T01:14:39.216 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-11492.g9b3739e7 1/1 2022-04-10T01:14:39.217 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:14:39.217 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2022-04-10T01:14:39.217 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-immutable-object-cache-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:14:39.217 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:14:39.217 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2022-04-10T01:14:39.375 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : cephadm-2:17.0.0-11492.g9b3739e7.el8.noarch 1/1 2022-04-10T01:14:39.376 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:14:39.376 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2022-04-10T01:14:39.376 INFO:teuthology.orchestra.run.smithi005.stdout: cephadm-2:17.0.0-11492.g9b3739e7.el8.noarch 2022-04-10T01:14:39.377 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:14:39.377 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2022-04-10T01:14:39.459 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph-mgr 2022-04-10T01:14:39.570 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-immutable-object-cache 2022-04-10T01:14:40.057 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:25 ago on Sun 10 Apr 2022 01:13:15 AM UTC. 2022-04-10T01:14:40.264 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:25 ago on Sun 10 Apr 2022 01:13:15 AM UTC. 2022-04-10T01:14:40.915 INFO:teuthology.orchestra.run.smithi043.stdout:Package ceph-mgr-2:17.0.0-11492.g9b3739e7.el8.x86_64 is already installed. 2022-04-10T01:14:40.972 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2022-04-10T01:14:40.973 INFO:teuthology.orchestra.run.smithi043.stdout:Nothing to do. 2022-04-10T01:14:40.973 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2022-04-10T01:14:41.046 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph-mgr-dashboard 2022-04-10T01:14:41.181 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2022-04-10T01:14:41.182 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2022-04-10T01:14:41.183 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repo Size 2022-04-10T01:14:41.183 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2022-04-10T01:14:41.183 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2022-04-10T01:14:41.183 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 169 k 2022-04-10T01:14:41.184 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:14:41.184 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2022-04-10T01:14:41.184 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2022-04-10T01:14:41.184 INFO:teuthology.orchestra.run.smithi005.stdout:Install 1 Package 2022-04-10T01:14:41.185 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:14:41.185 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 169 k 2022-04-10T01:14:41.185 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 567 k 2022-04-10T01:14:41.185 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2022-04-10T01:14:41.428 INFO:teuthology.orchestra.run.smithi005.stdout:ceph-immutable-object-cache-17.0.0-11492.g9b373 689 kB/s | 169 kB 00:00 2022-04-10T01:14:41.429 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2022-04-10T01:14:41.430 INFO:teuthology.orchestra.run.smithi005.stdout:Total 686 kB/s | 169 kB 00:00 2022-04-10T01:14:41.430 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2022-04-10T01:14:41.441 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2022-04-10T01:14:41.441 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2022-04-10T01:14:41.491 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2022-04-10T01:14:41.494 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2022-04-10T01:14:41.635 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:26 ago on Sun 10 Apr 2022 01:13:15 AM UTC. 2022-04-10T01:14:42.239 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2022-04-10T01:14:42.560 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2022-04-10T01:14:42.561 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2022-04-10T01:14:42.561 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repository Size 2022-04-10T01:14:42.561 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2022-04-10T01:14:42.562 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2022-04-10T01:14:42.562 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-dashboard noarch 2:17.0.0-11492.g9b3739e7.el8 ceph-noarch 1.6 M 2022-04-10T01:14:42.562 INFO:teuthology.orchestra.run.smithi043.stdout:Installing dependencies: 2022-04-10T01:14:42.562 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-grafana-dashboards noarch 2:17.0.0-11492.g9b3739e7.el8 ceph-noarch 23 k 2022-04-10T01:14:42.562 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-prometheus-alerts noarch 2:17.0.0-11492.g9b3739e7.el8 ceph-noarch 15 k 2022-04-10T01:14:42.563 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-Base 43 k 2022-04-10T01:14:42.563 INFO:teuthology.orchestra.run.smithi043.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2022-04-10T01:14:42.563 INFO:teuthology.orchestra.run.smithi043.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2022-04-10T01:14:42.563 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:14:42.564 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2022-04-10T01:14:42.564 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2022-04-10T01:14:42.564 INFO:teuthology.orchestra.run.smithi043.stdout:Install 6 Packages 2022-04-10T01:14:42.564 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:14:42.565 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 1.9 M 2022-04-10T01:14:42.565 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 7.5 M 2022-04-10T01:14:42.566 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2022-04-10T01:14:42.745 INFO:teuthology.orchestra.run.smithi043.stdout:(1/6): ceph-grafana-dashboards-17.0.0-11492.g9b 125 kB/s | 23 kB 00:00 2022-04-10T01:14:42.772 INFO:teuthology.orchestra.run.smithi043.stdout:(2/6): ceph-prometheus-alerts-17.0.0-11492.g9b3 70 kB/s | 15 kB 00:00 2022-04-10T01:14:42.823 INFO:teuthology.orchestra.run.smithi043.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 554 kB/s | 43 kB 00:00 2022-04-10T01:14:42.851 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-11492.g9b3739e7 1/1 2022-04-10T01:14:42.906 INFO:teuthology.orchestra.run.smithi043.stdout:(4/6): ceph-mgr-dashboard-17.0.0-11492.g9b3739e 4.7 MB/s | 1.6 MB 00:00 2022-04-10T01:14:42.931 INFO:teuthology.orchestra.run.smithi043.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 212 kB/s | 34 kB 00:00 2022-04-10T01:14:43.251 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-11492.g9b3739e7 1/1 2022-04-10T01:14:43.310 INFO:teuthology.orchestra.run.smithi043.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 403 kB/s | 196 kB 00:00 2022-04-10T01:14:43.312 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2022-04-10T01:14:43.313 INFO:teuthology.orchestra.run.smithi043.stdout:Total 2.6 MB/s | 1.9 MB 00:00 2022-04-10T01:14:43.325 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2022-04-10T01:14:43.335 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2022-04-10T01:14:43.335 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2022-04-10T01:14:43.631 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2022-04-10T01:14:43.635 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2022-04-10T01:14:43.664 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-11492.g9b3739e7 1/1 2022-04-10T01:14:43.664 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:14:43.665 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2022-04-10T01:14:43.665 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-immutable-object-cache-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:14:43.665 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:14:43.665 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2022-04-10T01:14:43.816 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-mgr 2022-04-10T01:14:44.413 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:29 ago on Sun 10 Apr 2022 01:13:15 AM UTC. 2022-04-10T01:14:44.500 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2022-04-10T01:14:44.702 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2022-04-10T01:14:44.946 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2022-04-10T01:14:45.153 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2022-04-10T01:14:45.262 INFO:teuthology.orchestra.run.smithi005.stdout:Package ceph-mgr-2:17.0.0-11492.g9b3739e7.el8.x86_64 is already installed. 2022-04-10T01:14:45.300 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-11492.g9b3739e7.el8. 4/6 2022-04-10T01:14:45.319 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2022-04-10T01:14:45.320 INFO:teuthology.orchestra.run.smithi005.stdout:Nothing to do. 2022-04-10T01:14:45.320 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2022-04-10T01:14:45.397 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-mgr-dashboard 2022-04-10T01:14:45.674 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-11492.g9b3739e7.el8 5/6 2022-04-10T01:14:45.923 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-11492.g9b3739e7.el8.noar 6/6 2022-04-10T01:14:45.986 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:30 ago on Sun 10 Apr 2022 01:13:15 AM UTC. 2022-04-10T01:14:46.175 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-11492.g9b3739e7.el8.noar 6/6 2022-04-10T01:14:46.175 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-11492.g9b3739e7.el8 1/6 2022-04-10T01:14:46.176 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-11492.g9b3739e7.el8.noar 2/6 2022-04-10T01:14:46.176 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-11492.g9b3739e7.el8. 3/6 2022-04-10T01:14:46.176 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2022-04-10T01:14:46.176 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2022-04-10T01:14:46.610 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2022-04-10T01:14:46.610 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:14:46.610 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2022-04-10T01:14:46.611 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-grafana-dashboards-2:17.0.0-11492.g9b3739e7.el8.noarch 2022-04-10T01:14:46.611 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-dashboard-2:17.0.0-11492.g9b3739e7.el8.noarch 2022-04-10T01:14:46.611 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-prometheus-alerts-2:17.0.0-11492.g9b3739e7.el8.noarch 2022-04-10T01:14:46.611 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jwt-1.6.1-2.el8.noarch 2022-04-10T01:14:46.612 INFO:teuthology.orchestra.run.smithi043.stdout: python3-repoze-lru-0.7-6.el8.noarch 2022-04-10T01:14:46.612 INFO:teuthology.orchestra.run.smithi043.stdout: python3-routes-2.4.1-12.el8.noarch 2022-04-10T01:14:46.612 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:14:46.612 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2022-04-10T01:14:46.722 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph-mgr-diskprediction-local 2022-04-10T01:14:46.897 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2022-04-10T01:14:46.898 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2022-04-10T01:14:46.899 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repository Size 2022-04-10T01:14:46.899 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2022-04-10T01:14:46.899 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2022-04-10T01:14:46.899 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-dashboard noarch 2:17.0.0-11492.g9b3739e7.el8 ceph-noarch 1.6 M 2022-04-10T01:14:46.900 INFO:teuthology.orchestra.run.smithi005.stdout:Installing dependencies: 2022-04-10T01:14:46.900 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-grafana-dashboards noarch 2:17.0.0-11492.g9b3739e7.el8 ceph-noarch 23 k 2022-04-10T01:14:46.900 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-prometheus-alerts noarch 2:17.0.0-11492.g9b3739e7.el8 ceph-noarch 15 k 2022-04-10T01:14:46.900 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-Base 43 k 2022-04-10T01:14:46.901 INFO:teuthology.orchestra.run.smithi005.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2022-04-10T01:14:46.901 INFO:teuthology.orchestra.run.smithi005.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2022-04-10T01:14:46.901 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:14:46.901 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2022-04-10T01:14:46.901 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2022-04-10T01:14:46.902 INFO:teuthology.orchestra.run.smithi005.stdout:Install 6 Packages 2022-04-10T01:14:46.902 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:14:46.903 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 1.9 M 2022-04-10T01:14:46.903 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 7.5 M 2022-04-10T01:14:46.903 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2022-04-10T01:14:47.059 INFO:teuthology.orchestra.run.smithi005.stdout:(1/6): ceph-grafana-dashboards-17.0.0-11492.g9b 143 kB/s | 23 kB 00:00 2022-04-10T01:14:47.084 INFO:teuthology.orchestra.run.smithi005.stdout:(2/6): ceph-prometheus-alerts-17.0.0-11492.g9b3 80 kB/s | 15 kB 00:00 2022-04-10T01:14:47.143 INFO:teuthology.orchestra.run.smithi005.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 509 kB/s | 43 kB 00:00 2022-04-10T01:14:47.193 INFO:teuthology.orchestra.run.smithi005.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 310 kB/s | 34 kB 00:00 2022-04-10T01:14:47.227 INFO:teuthology.orchestra.run.smithi005.stdout:(5/6): ceph-mgr-dashboard-17.0.0-11492.g9b3739e 5.0 MB/s | 1.6 MB 00:00 2022-04-10T01:14:47.310 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:32 ago on Sun 10 Apr 2022 01:13:15 AM UTC. 2022-04-10T01:14:47.394 INFO:teuthology.orchestra.run.smithi005.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 785 kB/s | 196 kB 00:00 2022-04-10T01:14:47.396 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2022-04-10T01:14:47.396 INFO:teuthology.orchestra.run.smithi005.stdout:Total 3.9 MB/s | 1.9 MB 00:00 2022-04-10T01:14:47.407 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2022-04-10T01:14:47.416 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2022-04-10T01:14:47.416 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2022-04-10T01:14:47.513 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2022-04-10T01:14:47.516 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2022-04-10T01:14:47.791 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2022-04-10T01:14:48.008 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2022-04-10T01:14:48.201 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2022-04-10T01:14:48.202 INFO:teuthology.orchestra.run.smithi043.stdout:===================================================================================================== 2022-04-10T01:14:48.203 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repository Size 2022-04-10T01:14:48.203 INFO:teuthology.orchestra.run.smithi043.stdout:===================================================================================================== 2022-04-10T01:14:48.203 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2022-04-10T01:14:48.203 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-11492.g9b3739e7.el8 ceph-noarch 7.4 M 2022-04-10T01:14:48.204 INFO:teuthology.orchestra.run.smithi043.stdout:Installing dependencies: 2022-04-10T01:14:48.204 INFO:teuthology.orchestra.run.smithi043.stdout: libgfortran x86_64 8.5.0-12.el8 CentOS-Base 644 k 2022-04-10T01:14:48.204 INFO:teuthology.orchestra.run.smithi043.stdout: libquadmath x86_64 8.5.0-12.el8 CentOS-Base 171 k 2022-04-10T01:14:48.204 INFO:teuthology.orchestra.run.smithi043.stdout: openblas x86_64 0.3.15-3.el8 CentOS-AppStream 4.7 M 2022-04-10T01:14:48.205 INFO:teuthology.orchestra.run.smithi043.stdout: openblas-threads x86_64 0.3.15-3.el8 CentOS-AppStream 4.9 M 2022-04-10T01:14:48.205 INFO:teuthology.orchestra.run.smithi043.stdout: python3-numpy x86_64 1:1.14.3-10.el8 CentOS-AppStream 3.7 M 2022-04-10T01:14:48.205 INFO:teuthology.orchestra.run.smithi043.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 CentOS-AppStream 225 k 2022-04-10T01:14:48.205 INFO:teuthology.orchestra.run.smithi043.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 CentOS-AppStream 14 M 2022-04-10T01:14:48.205 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:14:48.206 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2022-04-10T01:14:48.206 INFO:teuthology.orchestra.run.smithi043.stdout:===================================================================================================== 2022-04-10T01:14:48.206 INFO:teuthology.orchestra.run.smithi043.stdout:Install 8 Packages 2022-04-10T01:14:48.206 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:14:48.207 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 35 M 2022-04-10T01:14:48.208 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 210 M 2022-04-10T01:14:48.208 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2022-04-10T01:14:48.380 INFO:teuthology.orchestra.run.smithi043.stdout:(1/8): openblas-0.3.15-3.el8.x86_64.rpm 27 MB/s | 4.7 MB 00:00 2022-04-10T01:14:48.472 INFO:teuthology.orchestra.run.smithi043.stdout:(2/8): openblas-threads-0.3.15-3.el8.x86_64.rpm 18 MB/s | 4.9 MB 00:00 2022-04-10T01:14:48.506 INFO:teuthology.orchestra.run.smithi043.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 6.6 MB/s | 225 kB 00:00 2022-04-10T01:14:48.581 INFO:teuthology.orchestra.run.smithi043.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 19 MB/s | 3.7 MB 00:00 2022-04-10T01:14:48.631 INFO:teuthology.orchestra.run.smithi043.stdout:(5/8): libgfortran-8.5.0-12.el8.x86_64.rpm 13 MB/s | 644 kB 00:00 2022-04-10T01:14:48.665 INFO:teuthology.orchestra.run.smithi043.stdout:(6/8): libquadmath-8.5.0-12.el8.x86_64.rpm 5.0 MB/s | 171 kB 00:00 2022-04-10T01:14:48.774 INFO:teuthology.orchestra.run.smithi043.stdout:(7/8): ceph-mgr-diskprediction-local-17.0.0-114 13 MB/s | 7.4 MB 00:00 2022-04-10T01:14:48.960 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2022-04-10T01:14:48.991 INFO:teuthology.orchestra.run.smithi043.stdout:(8/8): python3-scipy-1.0.0-21.module_el8.5.0+77 28 MB/s | 14 MB 00:00 2022-04-10T01:14:48.993 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2022-04-10T01:14:48.993 INFO:teuthology.orchestra.run.smithi043.stdout:Total 45 MB/s | 35 MB 00:00 2022-04-10T01:14:49.229 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2022-04-10T01:14:49.248 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2022-04-10T01:14:49.249 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2022-04-10T01:14:49.611 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2022-04-10T01:14:49.614 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2022-04-10T01:14:49.617 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2022-04-10T01:14:49.774 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-11492.g9b3739e7.el8. 4/6 2022-04-10T01:14:50.139 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-11492.g9b3739e7.el8 5/6 2022-04-10T01:14:50.295 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2022-04-10T01:14:50.431 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-11492.g9b3739e7.el8.noar 6/6 2022-04-10T01:14:50.433 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libquadmath-8.5.0-12.el8.x86_64 1/8 2022-04-10T01:14:50.567 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: libquadmath-8.5.0-12.el8.x86_64 1/8 2022-04-10T01:14:50.668 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-11492.g9b3739e7.el8.noar 6/6 2022-04-10T01:14:50.668 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-11492.g9b3739e7.el8 1/6 2022-04-10T01:14:50.669 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-11492.g9b3739e7.el8.noar 2/6 2022-04-10T01:14:50.669 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-11492.g9b3739e7.el8. 3/6 2022-04-10T01:14:50.669 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2022-04-10T01:14:50.669 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2022-04-10T01:14:50.716 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libgfortran-8.5.0-12.el8.x86_64 2/8 2022-04-10T01:14:51.092 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2022-04-10T01:14:51.093 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:14:51.093 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2022-04-10T01:14:51.093 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-grafana-dashboards-2:17.0.0-11492.g9b3739e7.el8.noarch 2022-04-10T01:14:51.094 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-dashboard-2:17.0.0-11492.g9b3739e7.el8.noarch 2022-04-10T01:14:51.094 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-prometheus-alerts-2:17.0.0-11492.g9b3739e7.el8.noarch 2022-04-10T01:14:51.094 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jwt-1.6.1-2.el8.noarch 2022-04-10T01:14:51.094 INFO:teuthology.orchestra.run.smithi005.stdout: python3-repoze-lru-0.7-6.el8.noarch 2022-04-10T01:14:51.094 INFO:teuthology.orchestra.run.smithi005.stdout: python3-routes-2.4.1-12.el8.noarch 2022-04-10T01:14:51.095 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:14:51.095 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2022-04-10T01:14:51.203 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-mgr-diskprediction-local 2022-04-10T01:14:51.350 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: libgfortran-8.5.0-12.el8.x86_64 2/8 2022-04-10T01:14:51.557 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : openblas-0.3.15-3.el8.x86_64 3/8 2022-04-10T01:14:51.791 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:36 ago on Sun 10 Apr 2022 01:13:15 AM UTC. 2022-04-10T01:14:52.235 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 3/8 2022-04-10T01:14:52.407 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 4/8 2022-04-10T01:14:52.709 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2022-04-10T01:14:52.711 INFO:teuthology.orchestra.run.smithi005.stdout:===================================================================================================== 2022-04-10T01:14:52.711 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repository Size 2022-04-10T01:14:52.711 INFO:teuthology.orchestra.run.smithi005.stdout:===================================================================================================== 2022-04-10T01:14:52.711 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2022-04-10T01:14:52.712 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-11492.g9b3739e7.el8 ceph-noarch 7.4 M 2022-04-10T01:14:52.712 INFO:teuthology.orchestra.run.smithi005.stdout:Installing dependencies: 2022-04-10T01:14:52.712 INFO:teuthology.orchestra.run.smithi005.stdout: libgfortran x86_64 8.5.0-12.el8 CentOS-Base 644 k 2022-04-10T01:14:52.712 INFO:teuthology.orchestra.run.smithi005.stdout: libquadmath x86_64 8.5.0-12.el8 CentOS-Base 171 k 2022-04-10T01:14:52.713 INFO:teuthology.orchestra.run.smithi005.stdout: openblas x86_64 0.3.15-3.el8 CentOS-AppStream 4.7 M 2022-04-10T01:14:52.713 INFO:teuthology.orchestra.run.smithi005.stdout: openblas-threads x86_64 0.3.15-3.el8 CentOS-AppStream 4.9 M 2022-04-10T01:14:52.713 INFO:teuthology.orchestra.run.smithi005.stdout: python3-numpy x86_64 1:1.14.3-10.el8 CentOS-AppStream 3.7 M 2022-04-10T01:14:52.713 INFO:teuthology.orchestra.run.smithi005.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 CentOS-AppStream 225 k 2022-04-10T01:14:52.713 INFO:teuthology.orchestra.run.smithi005.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 CentOS-AppStream 14 M 2022-04-10T01:14:52.714 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:14:52.714 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2022-04-10T01:14:52.714 INFO:teuthology.orchestra.run.smithi005.stdout:===================================================================================================== 2022-04-10T01:14:52.714 INFO:teuthology.orchestra.run.smithi005.stdout:Install 8 Packages 2022-04-10T01:14:52.715 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:14:52.716 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 35 M 2022-04-10T01:14:52.716 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 210 M 2022-04-10T01:14:52.716 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2022-04-10T01:14:52.893 INFO:teuthology.orchestra.run.smithi005.stdout:(1/8): openblas-0.3.15-3.el8.x86_64.rpm 26 MB/s | 4.7 MB 00:00 2022-04-10T01:14:52.985 INFO:teuthology.orchestra.run.smithi005.stdout:(2/8): openblas-threads-0.3.15-3.el8.x86_64.rpm 18 MB/s | 4.9 MB 00:00 2022-04-10T01:14:53.018 INFO:teuthology.orchestra.run.smithi005.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 6.6 MB/s | 225 kB 00:00 2022-04-10T01:14:53.094 INFO:teuthology.orchestra.run.smithi005.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 19 MB/s | 3.7 MB 00:00 2022-04-10T01:14:53.146 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 4/8 2022-04-10T01:14:53.166 INFO:teuthology.orchestra.run.smithi005.stdout:(5/8): libgfortran-8.5.0-12.el8.x86_64.rpm 8.8 MB/s | 644 kB 00:00 2022-04-10T01:14:53.202 INFO:teuthology.orchestra.run.smithi005.stdout:(6/8): libquadmath-8.5.0-12.el8.x86_64.rpm 4.6 MB/s | 171 kB 00:00 2022-04-10T01:14:53.319 INFO:teuthology.orchestra.run.smithi005.stdout:(7/8): ceph-mgr-diskprediction-local-17.0.0-114 12 MB/s | 7.4 MB 00:00 2022-04-10T01:14:53.482 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2022-04-10T01:14:53.553 INFO:teuthology.orchestra.run.smithi005.stdout:(8/8): python3-scipy-1.0.0-21.module_el8.5.0+77 26 MB/s | 14 MB 00:00 2022-04-10T01:14:53.555 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2022-04-10T01:14:53.556 INFO:teuthology.orchestra.run.smithi005.stdout:Total 42 MB/s | 35 MB 00:00 2022-04-10T01:14:53.741 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2022-04-10T01:14:53.793 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2022-04-10T01:14:53.813 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2022-04-10T01:14:53.813 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2022-04-10T01:14:54.169 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2022-04-10T01:14:54.172 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2022-04-10T01:14:54.775 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2022-04-10T01:14:55.032 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libquadmath-8.5.0-12.el8.x86_64 1/8 2022-04-10T01:14:55.195 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: libquadmath-8.5.0-12.el8.x86_64 1/8 2022-04-10T01:14:55.331 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libgfortran-8.5.0-12.el8.x86_64 2/8 2022-04-10T01:14:55.623 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2022-04-10T01:14:55.996 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: libgfortran-8.5.0-12.el8.x86_64 2/8 2022-04-10T01:14:56.223 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : openblas-0.3.15-3.el8.x86_64 3/8 2022-04-10T01:14:56.874 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 3/8 2022-04-10T01:14:56.875 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2022-04-10T01:14:57.065 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 4/8 2022-04-10T01:14:57.090 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-11492.g9b3739 8/8 2022-04-10T01:14:57.372 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-11492.g9b3739 8/8 2022-04-10T01:14:57.373 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-11492.g9b3739 1/8 2022-04-10T01:14:57.373 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 2/8 2022-04-10T01:14:57.373 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 3/8 2022-04-10T01:14:57.373 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2022-04-10T01:14:57.374 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2022-04-10T01:14:57.374 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2022-04-10T01:14:57.374 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libgfortran-8.5.0-12.el8.x86_64 7/8 2022-04-10T01:14:57.785 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libquadmath-8.5.0-12.el8.x86_64 8/8 2022-04-10T01:14:57.786 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:14:57.786 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2022-04-10T01:14:57.786 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-diskprediction-local-2:17.0.0-11492.g9b3739e7.el8.noarch 2022-04-10T01:14:57.787 INFO:teuthology.orchestra.run.smithi043.stdout: libgfortran-8.5.0-12.el8.x86_64 2022-04-10T01:14:57.787 INFO:teuthology.orchestra.run.smithi043.stdout: libquadmath-8.5.0-12.el8.x86_64 2022-04-10T01:14:57.787 INFO:teuthology.orchestra.run.smithi043.stdout: openblas-0.3.15-3.el8.x86_64 2022-04-10T01:14:57.787 INFO:teuthology.orchestra.run.smithi043.stdout: openblas-threads-0.3.15-3.el8.x86_64 2022-04-10T01:14:57.787 INFO:teuthology.orchestra.run.smithi043.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2022-04-10T01:14:57.788 INFO:teuthology.orchestra.run.smithi043.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2022-04-10T01:14:57.788 INFO:teuthology.orchestra.run.smithi043.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2022-04-10T01:14:57.788 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:14:57.788 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2022-04-10T01:14:57.802 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 4/8 2022-04-10T01:14:57.887 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph-mgr-rook 2022-04-10T01:14:58.189 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2022-04-10T01:14:58.464 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2022-04-10T01:14:58.475 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:43 ago on Sun 10 Apr 2022 01:13:15 AM UTC. 2022-04-10T01:14:59.391 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2022-04-10T01:14:59.393 INFO:teuthology.orchestra.run.smithi043.stdout:========================================================================================= 2022-04-10T01:14:59.394 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repository Size 2022-04-10T01:14:59.394 INFO:teuthology.orchestra.run.smithi043.stdout:========================================================================================= 2022-04-10T01:14:59.394 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2022-04-10T01:14:59.394 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-rook noarch 2:17.0.0-11492.g9b3739e7.el8 ceph-noarch 47 k 2022-04-10T01:14:59.395 INFO:teuthology.orchestra.run.smithi043.stdout:Installing dependencies: 2022-04-10T01:14:59.395 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2022-04-10T01:14:59.395 INFO:teuthology.orchestra.run.smithi043.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2022-04-10T01:14:59.395 INFO:teuthology.orchestra.run.smithi043.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2022-04-10T01:14:59.396 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2022-04-10T01:14:59.396 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2022-04-10T01:14:59.396 INFO:teuthology.orchestra.run.smithi043.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2022-04-10T01:14:59.396 INFO:teuthology.orchestra.run.smithi043.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-Base 155 k 2022-04-10T01:14:59.396 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2022-04-10T01:14:59.397 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2022-04-10T01:14:59.397 INFO:teuthology.orchestra.run.smithi043.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-Base 43 k 2022-04-10T01:14:59.397 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2022-04-10T01:14:59.397 INFO:teuthology.orchestra.run.smithi043.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2022-04-10T01:14:59.398 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:14:59.398 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2022-04-10T01:14:59.398 INFO:teuthology.orchestra.run.smithi043.stdout:========================================================================================= 2022-04-10T01:14:59.398 INFO:teuthology.orchestra.run.smithi043.stdout:Install 13 Packages 2022-04-10T01:14:59.399 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:14:59.399 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 1.9 M 2022-04-10T01:14:59.400 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 25 M 2022-04-10T01:14:59.400 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2022-04-10T01:14:59.457 INFO:teuthology.orchestra.run.smithi043.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 442 kB/s | 27 kB 00:00 2022-04-10T01:14:59.483 INFO:teuthology.orchestra.run.smithi043.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 234 kB/s | 20 kB 00:00 2022-04-10T01:14:59.508 INFO:teuthology.orchestra.run.smithi043.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 2.7 MB/s | 126 kB 00:00 2022-04-10T01:14:59.533 INFO:teuthology.orchestra.run.smithi043.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 2.1 MB/s | 110 kB 00:00 2022-04-10T01:14:59.559 INFO:teuthology.orchestra.run.smithi043.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 3.0 MB/s | 155 kB 00:00 2022-04-10T01:14:59.584 INFO:teuthology.orchestra.run.smithi043.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 851 kB/s | 43 kB 00:00 2022-04-10T01:14:59.618 INFO:teuthology.orchestra.run.smithi043.stdout:(7/13): ceph-mgr-rook-17.0.0-11492.g9b3739e7.el 213 kB/s | 47 kB 00:00 2022-04-10T01:14:59.818 INFO:teuthology.orchestra.run.smithi043.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 127 kB/s | 33 kB 00:00 2022-04-10T01:14:59.835 INFO:teuthology.orchestra.run.smithi043.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 65 kB/s | 16 kB 00:00 2022-04-10T01:14:59.860 INFO:teuthology.orchestra.run.smithi043.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 336 kB/s | 81 kB 00:00 2022-04-10T01:14:59.960 INFO:teuthology.orchestra.run.smithi043.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 612 kB/s | 61 kB 00:00 2022-04-10T01:15:00.019 INFO:teuthology.orchestra.run.smithi043.stdout:(12/13): python3-rsa-4.8-1.el8.noarch.rpm 335 kB/s | 61 kB 00:00 2022-04-10T01:15:00.338 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2022-04-10T01:15:00.378 INFO:teuthology.orchestra.run.smithi043.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 2.1 MB/s | 1.2 MB 00:00 2022-04-10T01:15:00.380 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2022-04-10T01:15:00.380 INFO:teuthology.orchestra.run.smithi043.stdout:Total 2.0 MB/s | 1.9 MB 00:00 2022-04-10T01:15:00.441 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2022-04-10T01:15:00.458 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2022-04-10T01:15:00.458 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2022-04-10T01:15:00.605 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2022-04-10T01:15:00.608 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2022-04-10T01:15:01.629 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2022-04-10T01:15:01.759 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2022-04-10T01:15:01.888 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2022-04-10T01:15:01.956 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-11492.g9b3739 8/8 2022-04-10T01:15:02.085 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2022-04-10T01:15:02.222 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-11492.g9b3739 8/8 2022-04-10T01:15:02.223 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-11492.g9b3739 1/8 2022-04-10T01:15:02.223 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 2/8 2022-04-10T01:15:02.223 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 3/8 2022-04-10T01:15:02.223 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2022-04-10T01:15:02.224 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2022-04-10T01:15:02.224 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2022-04-10T01:15:02.224 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libgfortran-8.5.0-12.el8.x86_64 7/8 2022-04-10T01:15:02.378 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-rsa-4.8-1.el8.noarch 3/13 2022-04-10T01:15:02.613 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libquadmath-8.5.0-12.el8.x86_64 8/8 2022-04-10T01:15:02.613 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:02.613 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2022-04-10T01:15:02.614 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-diskprediction-local-2:17.0.0-11492.g9b3739e7.el8.noarch 2022-04-10T01:15:02.614 INFO:teuthology.orchestra.run.smithi005.stdout: libgfortran-8.5.0-12.el8.x86_64 2022-04-10T01:15:02.614 INFO:teuthology.orchestra.run.smithi005.stdout: libquadmath-8.5.0-12.el8.x86_64 2022-04-10T01:15:02.614 INFO:teuthology.orchestra.run.smithi005.stdout: openblas-0.3.15-3.el8.x86_64 2022-04-10T01:15:02.615 INFO:teuthology.orchestra.run.smithi005.stdout: openblas-threads-0.3.15-3.el8.x86_64 2022-04-10T01:15:02.615 INFO:teuthology.orchestra.run.smithi005.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2022-04-10T01:15:02.615 INFO:teuthology.orchestra.run.smithi005.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2022-04-10T01:15:02.615 INFO:teuthology.orchestra.run.smithi005.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2022-04-10T01:15:02.616 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:02.616 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2022-04-10T01:15:02.616 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2022-04-10T01:15:02.705 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-mgr-rook 2022-04-10T01:15:02.831 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2022-04-10T01:15:03.050 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2022-04-10T01:15:03.296 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:48 ago on Sun 10 Apr 2022 01:13:15 AM UTC. 2022-04-10T01:15:03.344 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2022-04-10T01:15:03.547 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2022-04-10T01:15:04.197 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2022-04-10T01:15:04.199 INFO:teuthology.orchestra.run.smithi005.stdout:========================================================================================= 2022-04-10T01:15:04.199 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repository Size 2022-04-10T01:15:04.199 INFO:teuthology.orchestra.run.smithi005.stdout:========================================================================================= 2022-04-10T01:15:04.200 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2022-04-10T01:15:04.200 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-rook noarch 2:17.0.0-11492.g9b3739e7.el8 ceph-noarch 47 k 2022-04-10T01:15:04.200 INFO:teuthology.orchestra.run.smithi005.stdout:Installing dependencies: 2022-04-10T01:15:04.200 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2022-04-10T01:15:04.201 INFO:teuthology.orchestra.run.smithi005.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2022-04-10T01:15:04.201 INFO:teuthology.orchestra.run.smithi005.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2022-04-10T01:15:04.201 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2022-04-10T01:15:04.201 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2022-04-10T01:15:04.202 INFO:teuthology.orchestra.run.smithi005.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2022-04-10T01:15:04.202 INFO:teuthology.orchestra.run.smithi005.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-Base 155 k 2022-04-10T01:15:04.202 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2022-04-10T01:15:04.202 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2022-04-10T01:15:04.203 INFO:teuthology.orchestra.run.smithi005.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-Base 43 k 2022-04-10T01:15:04.203 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2022-04-10T01:15:04.203 INFO:teuthology.orchestra.run.smithi005.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2022-04-10T01:15:04.203 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:04.204 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2022-04-10T01:15:04.204 INFO:teuthology.orchestra.run.smithi005.stdout:========================================================================================= 2022-04-10T01:15:04.204 INFO:teuthology.orchestra.run.smithi005.stdout:Install 13 Packages 2022-04-10T01:15:04.204 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:04.205 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 1.9 M 2022-04-10T01:15:04.205 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 25 M 2022-04-10T01:15:04.206 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2022-04-10T01:15:04.270 INFO:teuthology.orchestra.run.smithi005.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 396 kB/s | 27 kB 00:00 2022-04-10T01:15:04.329 INFO:teuthology.orchestra.run.smithi005.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 158 kB/s | 20 kB 00:00 2022-04-10T01:15:04.354 INFO:teuthology.orchestra.run.smithi005.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.5 MB/s | 126 kB 00:00 2022-04-10T01:15:04.371 INFO:teuthology.orchestra.run.smithi005.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 2.6 MB/s | 110 kB 00:00 2022-04-10T01:15:04.396 INFO:teuthology.orchestra.run.smithi005.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 3.6 MB/s | 155 kB 00:00 2022-04-10T01:15:04.413 INFO:teuthology.orchestra.run.smithi005.stdout:(6/13): ceph-mgr-rook-17.0.0-11492.g9b3739e7.el 223 kB/s | 47 kB 00:00 2022-04-10T01:15:04.430 INFO:teuthology.orchestra.run.smithi005.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 721 kB/s | 43 kB 00:00 2022-04-10T01:15:04.514 INFO:teuthology.orchestra.run.smithi005.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 162 kB/s | 16 kB 00:00 2022-04-10T01:15:04.531 INFO:teuthology.orchestra.run.smithi005.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 246 kB/s | 33 kB 00:00 2022-04-10T01:15:04.565 INFO:teuthology.orchestra.run.smithi005.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 607 kB/s | 81 kB 00:00 2022-04-10T01:15:04.632 INFO:teuthology.orchestra.run.smithi005.stdout:(11/13): python3-rsa-4.8-1.el8.noarch.rpm 613 kB/s | 61 kB 00:00 2022-04-10T01:15:04.670 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2022-04-10T01:15:04.673 INFO:teuthology.orchestra.run.smithi005.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 564 kB/s | 61 kB 00:00 2022-04-10T01:15:04.929 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2022-04-10T01:15:05.113 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2022-04-10T01:15:05.124 INFO:teuthology.orchestra.run.smithi005.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 1.9 MB/s | 1.2 MB 00:00 2022-04-10T01:15:05.126 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2022-04-10T01:15:05.126 INFO:teuthology.orchestra.run.smithi005.stdout:Total 2.1 MB/s | 1.9 MB 00:00 2022-04-10T01:15:05.187 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2022-04-10T01:15:05.204 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2022-04-10T01:15:05.205 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2022-04-10T01:15:05.295 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2022-04-10T01:15:05.299 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2022-04-10T01:15:05.350 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2022-04-10T01:15:05.505 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-rook-2:17.0.0-11492.g9b3739e7.el8.noarch 13/13 2022-04-10T01:15:05.763 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-11492.g9b3739e7.el8.noarch 13/13 2022-04-10T01:15:05.763 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-rook-2:17.0.0-11492.g9b3739e7.el8.noarch 1/13 2022-04-10T01:15:05.764 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2022-04-10T01:15:05.764 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2022-04-10T01:15:05.764 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2022-04-10T01:15:05.764 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2022-04-10T01:15:05.765 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2022-04-10T01:15:05.765 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2022-04-10T01:15:05.765 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2022-04-10T01:15:05.765 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2022-04-10T01:15:05.765 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2022-04-10T01:15:05.766 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2022-04-10T01:15:05.766 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 12/13 2022-04-10T01:15:06.156 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2022-04-10T01:15:06.230 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2022-04-10T01:15:06.230 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:06.230 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2022-04-10T01:15:06.231 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-rook-2:17.0.0-11492.g9b3739e7.el8.noarch 2022-04-10T01:15:06.231 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cachetools-3.1.1-4.el8.noarch 2022-04-10T01:15:06.231 INFO:teuthology.orchestra.run.smithi043.stdout: python3-certifi-2018.10.15-7.el8.noarch 2022-04-10T01:15:06.231 INFO:teuthology.orchestra.run.smithi043.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2022-04-10T01:15:06.232 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jsonpatch-1.21-2.el8.noarch 2022-04-10T01:15:06.232 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jsonpointer-1.10-11.el8.noarch 2022-04-10T01:15:06.232 INFO:teuthology.orchestra.run.smithi043.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2022-04-10T01:15:06.232 INFO:teuthology.orchestra.run.smithi043.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2022-04-10T01:15:06.233 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2022-04-10T01:15:06.233 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2022-04-10T01:15:06.233 INFO:teuthology.orchestra.run.smithi043.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2022-04-10T01:15:06.233 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rsa-4.8-1.el8.noarch 2022-04-10T01:15:06.234 INFO:teuthology.orchestra.run.smithi043.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2022-04-10T01:15:06.234 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:06.234 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2022-04-10T01:15:06.423 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2022-04-10T01:15:06.429 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph-mgr-cephadm 2022-04-10T01:15:06.651 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2022-04-10T01:15:06.941 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-rsa-4.8-1.el8.noarch 3/13 2022-04-10T01:15:07.021 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:52 ago on Sun 10 Apr 2022 01:13:15 AM UTC. 2022-04-10T01:15:07.188 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2022-04-10T01:15:07.405 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2022-04-10T01:15:07.673 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2022-04-10T01:15:07.923 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2022-04-10T01:15:07.924 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2022-04-10T01:15:07.924 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repository Size 2022-04-10T01:15:07.925 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2022-04-10T01:15:07.925 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2022-04-10T01:15:07.925 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-cephadm noarch 2:17.0.0-11492.g9b3739e7.el8 ceph-noarch 105 k 2022-04-10T01:15:07.926 INFO:teuthology.orchestra.run.smithi043.stdout:Installing dependencies: 2022-04-10T01:15:07.926 INFO:teuthology.orchestra.run.smithi043.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2022-04-10T01:15:07.927 INFO:teuthology.orchestra.run.smithi043.stdout: python3-babel noarch 2.5.1-7.el8 CentOS-AppStream 4.8 M 2022-04-10T01:15:07.927 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jinja2 noarch 2.10.1-3.el8 CentOS-AppStream 538 k 2022-04-10T01:15:07.928 INFO:teuthology.orchestra.run.smithi043.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2022-04-10T01:15:07.928 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:07.928 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2022-04-10T01:15:07.929 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2022-04-10T01:15:07.929 INFO:teuthology.orchestra.run.smithi043.stdout:Install 5 Packages 2022-04-10T01:15:07.930 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:07.931 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 5.9 M 2022-04-10T01:15:07.931 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 26 M 2022-04-10T01:15:07.932 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2022-04-10T01:15:07.985 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2022-04-10T01:15:08.003 INFO:teuthology.orchestra.run.smithi043.stdout:(1/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 6.9 MB/s | 538 kB 00:00 2022-04-10T01:15:08.120 INFO:teuthology.orchestra.run.smithi043.stdout:(2/5): python3-babel-2.5.1-7.el8.noarch.rpm 25 MB/s | 4.8 MB 00:00 2022-04-10T01:15:08.145 INFO:teuthology.orchestra.run.smithi043.stdout:(3/5): ceph-mgr-cephadm-17.0.0-11492.g9b3739e7. 480 kB/s | 105 kB 00:00 2022-04-10T01:15:08.212 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2022-04-10T01:15:08.262 INFO:teuthology.orchestra.run.smithi043.stdout:(4/5): python3-natsort-7.1.1-2.el8.noarch.rpm 425 kB/s | 60 kB 00:00 2022-04-10T01:15:08.488 INFO:teuthology.orchestra.run.smithi043.stdout:(5/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 983 kB/s | 476 kB 00:00 2022-04-10T01:15:08.489 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2022-04-10T01:15:08.490 INFO:teuthology.orchestra.run.smithi043.stdout:Total 11 MB/s | 5.9 MB 00:00 2022-04-10T01:15:08.542 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2022-04-10T01:15:08.554 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2022-04-10T01:15:08.554 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2022-04-10T01:15:08.655 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2022-04-10T01:15:08.658 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2022-04-10T01:15:08.917 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2022-04-10T01:15:09.278 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2022-04-10T01:15:09.294 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2022-04-10T01:15:09.611 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2022-04-10T01:15:09.812 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2022-04-10T01:15:10.050 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2022-04-10T01:15:10.213 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mgr-rook-2:17.0.0-11492.g9b3739e7.el8.noarch 13/13 2022-04-10T01:15:10.277 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2022-04-10T01:15:10.471 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-11492.g9b3739e7.el8.noarch 13/13 2022-04-10T01:15:10.471 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mgr-rook-2:17.0.0-11492.g9b3739e7.el8.noarch 1/13 2022-04-10T01:15:10.472 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2022-04-10T01:15:10.472 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2022-04-10T01:15:10.472 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2022-04-10T01:15:10.472 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2022-04-10T01:15:10.473 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2022-04-10T01:15:10.473 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2022-04-10T01:15:10.473 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2022-04-10T01:15:10.473 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2022-04-10T01:15:10.474 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2022-04-10T01:15:10.474 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2022-04-10T01:15:10.474 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 12/13 2022-04-10T01:15:10.644 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2022-04-10T01:15:10.882 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2022-04-10T01:15:10.930 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2022-04-10T01:15:10.930 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:10.931 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2022-04-10T01:15:10.931 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-rook-2:17.0.0-11492.g9b3739e7.el8.noarch 2022-04-10T01:15:10.931 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cachetools-3.1.1-4.el8.noarch 2022-04-10T01:15:10.931 INFO:teuthology.orchestra.run.smithi005.stdout: python3-certifi-2018.10.15-7.el8.noarch 2022-04-10T01:15:10.932 INFO:teuthology.orchestra.run.smithi005.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2022-04-10T01:15:10.932 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jsonpatch-1.21-2.el8.noarch 2022-04-10T01:15:10.932 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jsonpointer-1.10-11.el8.noarch 2022-04-10T01:15:10.932 INFO:teuthology.orchestra.run.smithi005.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2022-04-10T01:15:10.933 INFO:teuthology.orchestra.run.smithi005.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2022-04-10T01:15:10.933 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2022-04-10T01:15:10.933 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2022-04-10T01:15:10.933 INFO:teuthology.orchestra.run.smithi005.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2022-04-10T01:15:10.934 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rsa-4.8-1.el8.noarch 2022-04-10T01:15:10.934 INFO:teuthology.orchestra.run.smithi005.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2022-04-10T01:15:10.934 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:10.934 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2022-04-10T01:15:11.067 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-mgr-cephadm 2022-04-10T01:15:11.104 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-11492.g9b3739e7.el8.noarch 5/5 2022-04-10T01:15:11.360 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-11492.g9b3739e7.el8.noarch 5/5 2022-04-10T01:15:11.361 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-11492.g9b3739e7.el8.noarch 1/5 2022-04-10T01:15:11.361 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2022-04-10T01:15:11.361 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2022-04-10T01:15:11.362 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2022-04-10T01:15:11.658 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:56 ago on Sun 10 Apr 2022 01:13:15 AM UTC. 2022-04-10T01:15:11.768 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2022-04-10T01:15:11.769 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:11.769 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2022-04-10T01:15:11.769 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-cephadm-2:17.0.0-11492.g9b3739e7.el8.noarch 2022-04-10T01:15:11.769 INFO:teuthology.orchestra.run.smithi043.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2022-04-10T01:15:11.770 INFO:teuthology.orchestra.run.smithi043.stdout: python3-babel-2.5.1-7.el8.noarch 2022-04-10T01:15:11.770 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jinja2-2.10.1-3.el8.noarch 2022-04-10T01:15:11.770 INFO:teuthology.orchestra.run.smithi043.stdout: python3-natsort-7.1.1-2.el8.noarch 2022-04-10T01:15:11.771 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:11.771 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2022-04-10T01:15:11.892 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph-fuse 2022-04-10T01:15:12.480 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:57 ago on Sun 10 Apr 2022 01:13:15 AM UTC. 2022-04-10T01:15:12.561 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2022-04-10T01:15:12.563 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2022-04-10T01:15:12.563 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repository Size 2022-04-10T01:15:12.564 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2022-04-10T01:15:12.564 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2022-04-10T01:15:12.565 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-cephadm noarch 2:17.0.0-11492.g9b3739e7.el8 ceph-noarch 105 k 2022-04-10T01:15:12.565 INFO:teuthology.orchestra.run.smithi005.stdout:Installing dependencies: 2022-04-10T01:15:12.565 INFO:teuthology.orchestra.run.smithi005.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2022-04-10T01:15:12.565 INFO:teuthology.orchestra.run.smithi005.stdout: python3-babel noarch 2.5.1-7.el8 CentOS-AppStream 4.8 M 2022-04-10T01:15:12.566 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jinja2 noarch 2.10.1-3.el8 CentOS-AppStream 538 k 2022-04-10T01:15:12.566 INFO:teuthology.orchestra.run.smithi005.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2022-04-10T01:15:12.567 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:12.567 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2022-04-10T01:15:12.567 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2022-04-10T01:15:12.567 INFO:teuthology.orchestra.run.smithi005.stdout:Install 5 Packages 2022-04-10T01:15:12.568 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:12.568 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 5.9 M 2022-04-10T01:15:12.568 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 26 M 2022-04-10T01:15:12.569 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2022-04-10T01:15:12.642 INFO:teuthology.orchestra.run.smithi005.stdout:(1/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 6.8 MB/s | 538 kB 00:00 2022-04-10T01:15:12.751 INFO:teuthology.orchestra.run.smithi005.stdout:(2/5): python3-babel-2.5.1-7.el8.noarch.rpm 26 MB/s | 4.8 MB 00:00 2022-04-10T01:15:12.785 INFO:teuthology.orchestra.run.smithi005.stdout:(3/5): ceph-mgr-cephadm-17.0.0-11492.g9b3739e7. 479 kB/s | 105 kB 00:00 2022-04-10T01:15:12.919 INFO:teuthology.orchestra.run.smithi005.stdout:(4/5): python3-natsort-7.1.1-2.el8.noarch.rpm 361 kB/s | 60 kB 00:00 2022-04-10T01:15:13.118 INFO:teuthology.orchestra.run.smithi005.stdout:(5/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 1.0 MB/s | 476 kB 00:00 2022-04-10T01:15:13.120 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2022-04-10T01:15:13.120 INFO:teuthology.orchestra.run.smithi005.stdout:Total 11 MB/s | 5.9 MB 00:00 2022-04-10T01:15:13.176 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2022-04-10T01:15:13.187 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2022-04-10T01:15:13.188 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2022-04-10T01:15:13.289 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2022-04-10T01:15:13.292 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2022-04-10T01:15:13.367 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2022-04-10T01:15:13.369 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2022-04-10T01:15:13.369 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repository Size 2022-04-10T01:15:13.369 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2022-04-10T01:15:13.370 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2022-04-10T01:15:13.370 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-fuse x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 880 k 2022-04-10T01:15:13.370 INFO:teuthology.orchestra.run.smithi043.stdout:Installing dependencies: 2022-04-10T01:15:13.370 INFO:teuthology.orchestra.run.smithi043.stdout: fuse x86_64 2.9.7-15.el8 CentOS-Base 83 k 2022-04-10T01:15:13.371 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:13.371 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2022-04-10T01:15:13.371 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2022-04-10T01:15:13.371 INFO:teuthology.orchestra.run.smithi043.stdout:Install 2 Packages 2022-04-10T01:15:13.372 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:13.372 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 963 k 2022-04-10T01:15:13.372 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 3.1 M 2022-04-10T01:15:13.372 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2022-04-10T01:15:13.438 INFO:teuthology.orchestra.run.smithi043.stdout:(1/2): fuse-2.9.7-15.el8.x86_64.rpm 1.2 MB/s | 83 kB 00:00 2022-04-10T01:15:13.552 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2022-04-10T01:15:13.646 INFO:teuthology.orchestra.run.smithi043.stdout:(2/2): ceph-fuse-17.0.0-11492.g9b3739e7.el8.x86 3.1 MB/s | 880 kB 00:00 2022-04-10T01:15:13.647 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2022-04-10T01:15:13.648 INFO:teuthology.orchestra.run.smithi043.stdout:Total 3.4 MB/s | 963 kB 00:00 2022-04-10T01:15:13.658 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2022-04-10T01:15:13.680 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2022-04-10T01:15:13.680 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2022-04-10T01:15:13.759 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2022-04-10T01:15:13.763 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2022-04-10T01:15:13.885 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2022-04-10T01:15:13.961 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2022-04-10T01:15:14.236 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : fuse-2.9.7-15.el8.x86_64 1/2 2022-04-10T01:15:14.462 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-fuse-2:17.0.0-11492.g9b3739e7.el8.x86_64 2/2 2022-04-10T01:15:14.819 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-fuse-2:17.0.0-11492.g9b3739e7.el8.x86_64 2/2 2022-04-10T01:15:14.820 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-fuse-2:17.0.0-11492.g9b3739e7.el8.x86_64 1/2 2022-04-10T01:15:14.871 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2022-04-10T01:15:15.163 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 2/2 2022-04-10T01:15:15.164 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:15.164 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2022-04-10T01:15:15.164 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-fuse-2:17.0.0-11492.g9b3739e7.el8.x86_64 fuse-2.9.7-15.el8.x86_64 2022-04-10T01:15:15.164 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:15.165 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2022-04-10T01:15:15.241 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2022-04-10T01:15:15.395 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph-volume 2022-04-10T01:15:15.514 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2022-04-10T01:15:15.712 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-11492.g9b3739e7.el8.noarch 5/5 2022-04-10T01:15:15.965 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-11492.g9b3739e7.el8.noarch 5/5 2022-04-10T01:15:15.965 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-11492.g9b3739e7.el8.noarch 1/5 2022-04-10T01:15:15.966 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2022-04-10T01:15:15.966 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2022-04-10T01:15:15.966 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2022-04-10T01:15:15.997 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:02:00 ago on Sun 10 Apr 2022 01:13:15 AM UTC. 2022-04-10T01:15:16.360 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2022-04-10T01:15:16.360 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:16.361 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2022-04-10T01:15:16.361 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-cephadm-2:17.0.0-11492.g9b3739e7.el8.noarch 2022-04-10T01:15:16.361 INFO:teuthology.orchestra.run.smithi005.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2022-04-10T01:15:16.361 INFO:teuthology.orchestra.run.smithi005.stdout: python3-babel-2.5.1-7.el8.noarch 2022-04-10T01:15:16.362 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jinja2-2.10.1-3.el8.noarch 2022-04-10T01:15:16.362 INFO:teuthology.orchestra.run.smithi005.stdout: python3-natsort-7.1.1-2.el8.noarch 2022-04-10T01:15:16.362 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:16.362 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2022-04-10T01:15:16.476 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-fuse 2022-04-10T01:15:16.901 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2022-04-10T01:15:16.903 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2022-04-10T01:15:16.903 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repository Size 2022-04-10T01:15:16.903 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2022-04-10T01:15:16.903 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2022-04-10T01:15:16.904 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-volume noarch 2:17.0.0-11492.g9b3739e7.el8 ceph-noarch 259 k 2022-04-10T01:15:16.904 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:16.904 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2022-04-10T01:15:16.904 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2022-04-10T01:15:16.905 INFO:teuthology.orchestra.run.smithi043.stdout:Install 1 Package 2022-04-10T01:15:16.905 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:16.905 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 259 k 2022-04-10T01:15:16.905 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 1.3 M 2022-04-10T01:15:16.905 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2022-04-10T01:15:17.067 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:02:02 ago on Sun 10 Apr 2022 01:13:15 AM UTC. 2022-04-10T01:15:17.117 INFO:teuthology.orchestra.run.smithi043.stdout:ceph-volume-17.0.0-11492.g9b3739e7.el8.noarch.r 1.2 MB/s | 259 kB 00:00 2022-04-10T01:15:17.118 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2022-04-10T01:15:17.118 INFO:teuthology.orchestra.run.smithi043.stdout:Total 1.2 MB/s | 259 kB 00:00 2022-04-10T01:15:17.118 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2022-04-10T01:15:17.129 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2022-04-10T01:15:17.129 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2022-04-10T01:15:17.149 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2022-04-10T01:15:17.152 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2022-04-10T01:15:17.326 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2022-04-10T01:15:17.620 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-volume-2:17.0.0-11492.g9b3739e7.el8.noarch 1/1 2022-04-10T01:15:17.949 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2022-04-10T01:15:17.950 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2022-04-10T01:15:17.950 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repository Size 2022-04-10T01:15:17.950 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2022-04-10T01:15:17.951 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2022-04-10T01:15:17.951 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-fuse x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 880 k 2022-04-10T01:15:17.951 INFO:teuthology.orchestra.run.smithi005.stdout:Installing dependencies: 2022-04-10T01:15:17.951 INFO:teuthology.orchestra.run.smithi005.stdout: fuse x86_64 2.9.7-15.el8 CentOS-Base 83 k 2022-04-10T01:15:17.952 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:17.952 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2022-04-10T01:15:17.952 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2022-04-10T01:15:17.952 INFO:teuthology.orchestra.run.smithi005.stdout:Install 2 Packages 2022-04-10T01:15:17.953 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:17.953 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 963 k 2022-04-10T01:15:17.953 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 3.1 M 2022-04-10T01:15:17.953 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2022-04-10T01:15:18.018 INFO:teuthology.orchestra.run.smithi005.stdout:(1/2): fuse-2.9.7-15.el8.x86_64.rpm 1.2 MB/s | 83 kB 00:00 2022-04-10T01:15:18.037 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-volume-2:17.0.0-11492.g9b3739e7.el8.noarch 1/1 2022-04-10T01:15:18.235 INFO:teuthology.orchestra.run.smithi005.stdout:(2/2): ceph-fuse-17.0.0-11492.g9b3739e7.el8.x86 3.0 MB/s | 880 kB 00:00 2022-04-10T01:15:18.236 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2022-04-10T01:15:18.237 INFO:teuthology.orchestra.run.smithi005.stdout:Total 3.3 MB/s | 963 kB 00:00 2022-04-10T01:15:18.247 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2022-04-10T01:15:18.269 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2022-04-10T01:15:18.269 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2022-04-10T01:15:18.346 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2022-04-10T01:15:18.349 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2022-04-10T01:15:18.376 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-volume-2:17.0.0-11492.g9b3739e7.el8.noarch 1/1 2022-04-10T01:15:18.377 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:18.377 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2022-04-10T01:15:18.377 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-volume-2:17.0.0-11492.g9b3739e7.el8.noarch 2022-04-10T01:15:18.378 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:18.378 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2022-04-10T01:15:18.510 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2022-04-10T01:15:18.591 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install librados-devel 2022-04-10T01:15:18.802 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : fuse-2.9.7-15.el8.x86_64 1/2 2022-04-10T01:15:19.002 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-fuse-2:17.0.0-11492.g9b3739e7.el8.x86_64 2/2 2022-04-10T01:15:19.187 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:02:04 ago on Sun 10 Apr 2022 01:13:15 AM UTC. 2022-04-10T01:15:19.353 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-fuse-2:17.0.0-11492.g9b3739e7.el8.x86_64 2/2 2022-04-10T01:15:19.353 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-fuse-2:17.0.0-11492.g9b3739e7.el8.x86_64 1/2 2022-04-10T01:15:19.727 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 2/2 2022-04-10T01:15:19.727 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:19.727 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2022-04-10T01:15:19.728 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-fuse-2:17.0.0-11492.g9b3739e7.el8.x86_64 fuse-2.9.7-15.el8.x86_64 2022-04-10T01:15:19.728 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:19.728 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2022-04-10T01:15:19.942 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-volume 2022-04-10T01:15:20.082 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2022-04-10T01:15:20.083 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2022-04-10T01:15:20.083 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repo Size 2022-04-10T01:15:20.083 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2022-04-10T01:15:20.084 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2022-04-10T01:15:20.084 INFO:teuthology.orchestra.run.smithi043.stdout: librados-devel x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 125 k 2022-04-10T01:15:20.084 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:20.084 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2022-04-10T01:15:20.085 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2022-04-10T01:15:20.085 INFO:teuthology.orchestra.run.smithi043.stdout:Install 1 Package 2022-04-10T01:15:20.085 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:20.085 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 125 k 2022-04-10T01:15:20.085 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 431 k 2022-04-10T01:15:20.086 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2022-04-10T01:15:20.297 INFO:teuthology.orchestra.run.smithi043.stdout:librados-devel-17.0.0-11492.g9b3739e7.el8.x86_6 585 kB/s | 125 kB 00:00 2022-04-10T01:15:20.298 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2022-04-10T01:15:20.298 INFO:teuthology.orchestra.run.smithi043.stdout:Total 582 kB/s | 125 kB 00:00 2022-04-10T01:15:20.298 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2022-04-10T01:15:20.303 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2022-04-10T01:15:20.304 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2022-04-10T01:15:20.360 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2022-04-10T01:15:20.364 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2022-04-10T01:15:20.477 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2022-04-10T01:15:20.540 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:02:05 ago on Sun 10 Apr 2022 01:13:15 AM UTC. 2022-04-10T01:15:20.668 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : librados-devel-2:17.0.0-11492.g9b3739e7.el8.x86_64 1/1 2022-04-10T01:15:20.911 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librados-devel-2:17.0.0-11492.g9b3739e7.el8.x86_64 1/1 2022-04-10T01:15:21.243 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librados-devel-2:17.0.0-11492.g9b3739e7.el8.x86_64 1/1 2022-04-10T01:15:21.243 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:21.243 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2022-04-10T01:15:21.244 INFO:teuthology.orchestra.run.smithi043.stdout: librados-devel-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:15:21.244 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:21.244 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2022-04-10T01:15:21.413 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install libcephfs2 2022-04-10T01:15:21.441 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2022-04-10T01:15:21.442 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2022-04-10T01:15:21.442 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repository Size 2022-04-10T01:15:21.442 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2022-04-10T01:15:21.442 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2022-04-10T01:15:21.443 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-volume noarch 2:17.0.0-11492.g9b3739e7.el8 ceph-noarch 259 k 2022-04-10T01:15:21.443 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:21.443 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2022-04-10T01:15:21.443 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2022-04-10T01:15:21.444 INFO:teuthology.orchestra.run.smithi005.stdout:Install 1 Package 2022-04-10T01:15:21.444 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:21.444 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 259 k 2022-04-10T01:15:21.444 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 1.3 M 2022-04-10T01:15:21.445 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2022-04-10T01:15:21.663 INFO:teuthology.orchestra.run.smithi005.stdout:ceph-volume-17.0.0-11492.g9b3739e7.el8.noarch.r 1.1 MB/s | 259 kB 00:00 2022-04-10T01:15:21.664 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2022-04-10T01:15:21.665 INFO:teuthology.orchestra.run.smithi005.stdout:Total 1.1 MB/s | 259 kB 00:00 2022-04-10T01:15:21.665 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2022-04-10T01:15:21.675 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2022-04-10T01:15:21.676 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2022-04-10T01:15:21.699 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2022-04-10T01:15:21.702 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2022-04-10T01:15:21.885 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2022-04-10T01:15:22.009 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:02:07 ago on Sun 10 Apr 2022 01:13:15 AM UTC. 2022-04-10T01:15:22.445 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-volume-2:17.0.0-11492.g9b3739e7.el8.noarch 1/1 2022-04-10T01:15:22.852 INFO:teuthology.orchestra.run.smithi043.stdout:Package libcephfs2-2:17.0.0-11492.g9b3739e7.el8.x86_64 is already installed. 2022-04-10T01:15:22.868 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-volume-2:17.0.0-11492.g9b3739e7.el8.noarch 1/1 2022-04-10T01:15:22.910 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2022-04-10T01:15:22.912 INFO:teuthology.orchestra.run.smithi043.stdout:Nothing to do. 2022-04-10T01:15:22.912 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2022-04-10T01:15:23.264 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install libcephfs-devel 2022-04-10T01:15:23.299 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-volume-2:17.0.0-11492.g9b3739e7.el8.noarch 1/1 2022-04-10T01:15:23.300 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:23.300 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2022-04-10T01:15:23.300 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-volume-2:17.0.0-11492.g9b3739e7.el8.noarch 2022-04-10T01:15:23.300 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:23.301 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2022-04-10T01:15:23.509 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install librados-devel 2022-04-10T01:15:23.856 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:02:08 ago on Sun 10 Apr 2022 01:13:15 AM UTC. 2022-04-10T01:15:24.108 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:02:09 ago on Sun 10 Apr 2022 01:13:15 AM UTC. 2022-04-10T01:15:24.773 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2022-04-10T01:15:24.774 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2022-04-10T01:15:24.775 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repo Size 2022-04-10T01:15:24.775 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2022-04-10T01:15:24.775 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2022-04-10T01:15:24.775 INFO:teuthology.orchestra.run.smithi043.stdout: libcephfs-devel x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 25 k 2022-04-10T01:15:24.776 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:24.776 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2022-04-10T01:15:24.776 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2022-04-10T01:15:24.776 INFO:teuthology.orchestra.run.smithi043.stdout:Install 1 Package 2022-04-10T01:15:24.776 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:24.777 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 25 k 2022-04-10T01:15:24.777 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 109 k 2022-04-10T01:15:24.777 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2022-04-10T01:15:24.934 INFO:teuthology.orchestra.run.smithi043.stdout:libcephfs-devel-17.0.0-11492.g9b3739e7.el8.x86_ 156 kB/s | 25 kB 00:00 2022-04-10T01:15:24.935 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2022-04-10T01:15:24.936 INFO:teuthology.orchestra.run.smithi043.stdout:Total 155 kB/s | 25 kB 00:00 2022-04-10T01:15:24.936 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2022-04-10T01:15:24.938 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2022-04-10T01:15:24.939 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2022-04-10T01:15:24.940 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2022-04-10T01:15:24.943 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2022-04-10T01:15:24.991 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2022-04-10T01:15:24.992 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2022-04-10T01:15:24.992 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repo Size 2022-04-10T01:15:24.993 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2022-04-10T01:15:24.993 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2022-04-10T01:15:24.993 INFO:teuthology.orchestra.run.smithi005.stdout: librados-devel x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 125 k 2022-04-10T01:15:24.993 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:24.994 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2022-04-10T01:15:24.994 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2022-04-10T01:15:24.994 INFO:teuthology.orchestra.run.smithi005.stdout:Install 1 Package 2022-04-10T01:15:24.994 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:24.995 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 125 k 2022-04-10T01:15:24.995 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 431 k 2022-04-10T01:15:24.995 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2022-04-10T01:15:24.996 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2022-04-10T01:15:25.193 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libcephfs-devel-2:17.0.0-11492.g9b3739e7.el8.x86_64 1/1 2022-04-10T01:15:25.194 INFO:teuthology.orchestra.run.smithi005.stdout:librados-devel-17.0.0-11492.g9b3739e7.el8.x86_6 626 kB/s | 125 kB 00:00 2022-04-10T01:15:25.194 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2022-04-10T01:15:25.194 INFO:teuthology.orchestra.run.smithi005.stdout:Total 622 kB/s | 125 kB 00:00 2022-04-10T01:15:25.194 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2022-04-10T01:15:25.199 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2022-04-10T01:15:25.199 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2022-04-10T01:15:25.259 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2022-04-10T01:15:25.262 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2022-04-10T01:15:25.382 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2022-04-10T01:15:25.424 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: libcephfs-devel-2:17.0.0-11492.g9b3739e7.el8.x86_64 1/1 2022-04-10T01:15:25.584 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : librados-devel-2:17.0.0-11492.g9b3739e7.el8.x86_64 1/1 2022-04-10T01:15:25.750 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libcephfs-devel-2:17.0.0-11492.g9b3739e7.el8.x86_64 1/1 2022-04-10T01:15:25.751 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:25.751 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2022-04-10T01:15:25.751 INFO:teuthology.orchestra.run.smithi043.stdout: libcephfs-devel-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:15:25.751 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:25.751 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2022-04-10T01:15:25.857 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install librados2 2022-04-10T01:15:25.861 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: librados-devel-2:17.0.0-11492.g9b3739e7.el8.x86_64 1/1 2022-04-10T01:15:26.239 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librados-devel-2:17.0.0-11492.g9b3739e7.el8.x86_64 1/1 2022-04-10T01:15:26.240 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:26.240 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2022-04-10T01:15:26.240 INFO:teuthology.orchestra.run.smithi005.stdout: librados-devel-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:15:26.240 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:26.241 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2022-04-10T01:15:26.448 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:02:11 ago on Sun 10 Apr 2022 01:13:15 AM UTC. 2022-04-10T01:15:26.472 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install libcephfs2 2022-04-10T01:15:27.090 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:02:12 ago on Sun 10 Apr 2022 01:13:15 AM UTC. 2022-04-10T01:15:27.305 INFO:teuthology.orchestra.run.smithi043.stdout:Package librados2-2:17.0.0-11492.g9b3739e7.el8.x86_64 is already installed. 2022-04-10T01:15:27.363 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2022-04-10T01:15:27.365 INFO:teuthology.orchestra.run.smithi043.stdout:Nothing to do. 2022-04-10T01:15:27.365 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2022-04-10T01:15:27.454 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install librbd1 2022-04-10T01:15:27.938 INFO:teuthology.orchestra.run.smithi005.stdout:Package libcephfs2-2:17.0.0-11492.g9b3739e7.el8.x86_64 is already installed. 2022-04-10T01:15:27.997 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2022-04-10T01:15:27.999 INFO:teuthology.orchestra.run.smithi005.stdout:Nothing to do. 2022-04-10T01:15:27.999 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2022-04-10T01:15:28.040 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:02:13 ago on Sun 10 Apr 2022 01:13:15 AM UTC. 2022-04-10T01:15:28.077 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install libcephfs-devel 2022-04-10T01:15:28.669 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:02:13 ago on Sun 10 Apr 2022 01:13:15 AM UTC. 2022-04-10T01:15:28.901 INFO:teuthology.orchestra.run.smithi043.stdout:Package librbd1-2:17.0.0-11492.g9b3739e7.el8.x86_64 is already installed. 2022-04-10T01:15:28.959 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2022-04-10T01:15:28.960 INFO:teuthology.orchestra.run.smithi043.stdout:Nothing to do. 2022-04-10T01:15:28.960 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2022-04-10T01:15:29.231 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install python3-rados 2022-04-10T01:15:29.553 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2022-04-10T01:15:29.554 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2022-04-10T01:15:29.554 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repo Size 2022-04-10T01:15:29.554 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2022-04-10T01:15:29.555 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2022-04-10T01:15:29.555 INFO:teuthology.orchestra.run.smithi005.stdout: libcephfs-devel x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 25 k 2022-04-10T01:15:29.555 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:29.555 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2022-04-10T01:15:29.556 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2022-04-10T01:15:29.556 INFO:teuthology.orchestra.run.smithi005.stdout:Install 1 Package 2022-04-10T01:15:29.556 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:29.556 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 25 k 2022-04-10T01:15:29.556 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 109 k 2022-04-10T01:15:29.557 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2022-04-10T01:15:29.713 INFO:teuthology.orchestra.run.smithi005.stdout:libcephfs-devel-17.0.0-11492.g9b3739e7.el8.x86_ 157 kB/s | 25 kB 00:00 2022-04-10T01:15:29.714 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2022-04-10T01:15:29.714 INFO:teuthology.orchestra.run.smithi005.stdout:Total 156 kB/s | 25 kB 00:00 2022-04-10T01:15:29.715 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2022-04-10T01:15:29.717 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2022-04-10T01:15:29.718 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2022-04-10T01:15:29.720 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2022-04-10T01:15:29.723 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2022-04-10T01:15:29.777 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2022-04-10T01:15:29.823 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:02:14 ago on Sun 10 Apr 2022 01:13:15 AM UTC. 2022-04-10T01:15:29.975 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libcephfs-devel-2:17.0.0-11492.g9b3739e7.el8.x86_64 1/1 2022-04-10T01:15:30.195 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: libcephfs-devel-2:17.0.0-11492.g9b3739e7.el8.x86_64 1/1 2022-04-10T01:15:30.540 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libcephfs-devel-2:17.0.0-11492.g9b3739e7.el8.x86_64 1/1 2022-04-10T01:15:30.540 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:30.541 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2022-04-10T01:15:30.541 INFO:teuthology.orchestra.run.smithi005.stdout: libcephfs-devel-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:15:30.541 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:30.541 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2022-04-10T01:15:30.635 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install librados2 2022-04-10T01:15:30.662 INFO:teuthology.orchestra.run.smithi043.stdout:Package python3-rados-2:17.0.0-11492.g9b3739e7.el8.x86_64 is already installed. 2022-04-10T01:15:30.737 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2022-04-10T01:15:30.738 INFO:teuthology.orchestra.run.smithi043.stdout:Nothing to do. 2022-04-10T01:15:30.738 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2022-04-10T01:15:30.832 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install python3-rgw 2022-04-10T01:15:31.220 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:02:16 ago on Sun 10 Apr 2022 01:13:15 AM UTC. 2022-04-10T01:15:31.429 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:02:16 ago on Sun 10 Apr 2022 01:13:15 AM UTC. 2022-04-10T01:15:32.108 INFO:teuthology.orchestra.run.smithi005.stdout:Package librados2-2:17.0.0-11492.g9b3739e7.el8.x86_64 is already installed. 2022-04-10T01:15:32.166 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2022-04-10T01:15:32.167 INFO:teuthology.orchestra.run.smithi005.stdout:Nothing to do. 2022-04-10T01:15:32.168 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2022-04-10T01:15:32.252 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install librbd1 2022-04-10T01:15:32.257 INFO:teuthology.orchestra.run.smithi043.stdout:Package python3-rgw-2:17.0.0-11492.g9b3739e7.el8.x86_64 is already installed. 2022-04-10T01:15:32.317 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2022-04-10T01:15:32.318 INFO:teuthology.orchestra.run.smithi043.stdout:Nothing to do. 2022-04-10T01:15:32.318 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2022-04-10T01:15:32.411 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install python3-cephfs 2022-04-10T01:15:32.837 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:02:17 ago on Sun 10 Apr 2022 01:13:15 AM UTC. 2022-04-10T01:15:33.001 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:02:17 ago on Sun 10 Apr 2022 01:13:15 AM UTC. 2022-04-10T01:15:33.702 INFO:teuthology.orchestra.run.smithi005.stdout:Package librbd1-2:17.0.0-11492.g9b3739e7.el8.x86_64 is already installed. 2022-04-10T01:15:33.760 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2022-04-10T01:15:33.761 INFO:teuthology.orchestra.run.smithi005.stdout:Nothing to do. 2022-04-10T01:15:33.762 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2022-04-10T01:15:33.860 INFO:teuthology.orchestra.run.smithi043.stdout:Package python3-cephfs-2:17.0.0-11492.g9b3739e7.el8.x86_64 is already installed. 2022-04-10T01:15:33.918 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2022-04-10T01:15:33.919 INFO:teuthology.orchestra.run.smithi043.stdout:Nothing to do. 2022-04-10T01:15:33.920 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2022-04-10T01:15:34.037 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install python3-rbd 2022-04-10T01:15:34.186 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install python3-rados 2022-04-10T01:15:34.627 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:02:19 ago on Sun 10 Apr 2022 01:13:15 AM UTC. 2022-04-10T01:15:34.777 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:02:19 ago on Sun 10 Apr 2022 01:13:15 AM UTC. 2022-04-10T01:15:35.522 INFO:teuthology.orchestra.run.smithi043.stdout:Package python3-rbd-2:17.0.0-11492.g9b3739e7.el8.x86_64 is already installed. 2022-04-10T01:15:35.583 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2022-04-10T01:15:35.585 INFO:teuthology.orchestra.run.smithi043.stdout:Nothing to do. 2022-04-10T01:15:35.585 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2022-04-10T01:15:35.640 INFO:teuthology.orchestra.run.smithi005.stdout:Package python3-rados-2:17.0.0-11492.g9b3739e7.el8.x86_64 is already installed. 2022-04-10T01:15:35.668 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install rbd-fuse 2022-04-10T01:15:35.699 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2022-04-10T01:15:35.700 INFO:teuthology.orchestra.run.smithi005.stdout:Nothing to do. 2022-04-10T01:15:35.700 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2022-04-10T01:15:35.781 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install python3-rgw 2022-04-10T01:15:36.259 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:02:21 ago on Sun 10 Apr 2022 01:13:15 AM UTC. 2022-04-10T01:15:36.373 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:02:21 ago on Sun 10 Apr 2022 01:13:15 AM UTC. 2022-04-10T01:15:37.175 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2022-04-10T01:15:37.176 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2022-04-10T01:15:37.177 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repo Size 2022-04-10T01:15:37.177 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2022-04-10T01:15:37.177 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2022-04-10T01:15:37.177 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-fuse x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 93 k 2022-04-10T01:15:37.178 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:37.178 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2022-04-10T01:15:37.178 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2022-04-10T01:15:37.178 INFO:teuthology.orchestra.run.smithi043.stdout:Install 1 Package 2022-04-10T01:15:37.178 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:37.179 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 93 k 2022-04-10T01:15:37.179 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 280 k 2022-04-10T01:15:37.179 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2022-04-10T01:15:37.213 INFO:teuthology.orchestra.run.smithi005.stdout:Package python3-rgw-2:17.0.0-11492.g9b3739e7.el8.x86_64 is already installed. 2022-04-10T01:15:37.272 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2022-04-10T01:15:37.273 INFO:teuthology.orchestra.run.smithi005.stdout:Nothing to do. 2022-04-10T01:15:37.273 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2022-04-10T01:15:37.351 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install python3-cephfs 2022-04-10T01:15:37.374 INFO:teuthology.orchestra.run.smithi043.stdout:rbd-fuse-17.0.0-11492.g9b3739e7.el8.x86_64.rpm 473 kB/s | 93 kB 00:00 2022-04-10T01:15:37.375 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2022-04-10T01:15:37.376 INFO:teuthology.orchestra.run.smithi043.stdout:Total 470 kB/s | 93 kB 00:00 2022-04-10T01:15:37.376 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2022-04-10T01:15:37.386 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2022-04-10T01:15:37.386 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2022-04-10T01:15:37.439 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2022-04-10T01:15:37.442 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2022-04-10T01:15:37.547 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2022-04-10T01:15:37.766 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : rbd-fuse-2:17.0.0-11492.g9b3739e7.el8.x86_64 1/1 2022-04-10T01:15:37.942 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:02:22 ago on Sun 10 Apr 2022 01:13:15 AM UTC. 2022-04-10T01:15:38.045 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: rbd-fuse-2:17.0.0-11492.g9b3739e7.el8.x86_64 1/1 2022-04-10T01:15:38.417 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : rbd-fuse-2:17.0.0-11492.g9b3739e7.el8.x86_64 1/1 2022-04-10T01:15:38.418 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:38.418 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2022-04-10T01:15:38.418 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-fuse-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:15:38.419 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:38.419 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2022-04-10T01:15:38.609 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install rbd-mirror 2022-04-10T01:15:38.804 INFO:teuthology.orchestra.run.smithi005.stdout:Package python3-cephfs-2:17.0.0-11492.g9b3739e7.el8.x86_64 is already installed. 2022-04-10T01:15:38.863 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2022-04-10T01:15:38.864 INFO:teuthology.orchestra.run.smithi005.stdout:Nothing to do. 2022-04-10T01:15:38.864 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2022-04-10T01:15:38.945 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install python3-rbd 2022-04-10T01:15:39.205 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:02:24 ago on Sun 10 Apr 2022 01:13:15 AM UTC. 2022-04-10T01:15:39.545 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:02:24 ago on Sun 10 Apr 2022 01:13:15 AM UTC. 2022-04-10T01:15:40.107 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2022-04-10T01:15:40.108 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2022-04-10T01:15:40.108 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repo Size 2022-04-10T01:15:40.109 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2022-04-10T01:15:40.109 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2022-04-10T01:15:40.109 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-mirror x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 3.8 M 2022-04-10T01:15:40.109 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:40.109 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2022-04-10T01:15:40.110 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2022-04-10T01:15:40.110 INFO:teuthology.orchestra.run.smithi043.stdout:Install 1 Package 2022-04-10T01:15:40.110 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:40.110 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 3.8 M 2022-04-10T01:15:40.111 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 19 M 2022-04-10T01:15:40.111 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2022-04-10T01:15:40.391 INFO:teuthology.orchestra.run.smithi005.stdout:Package python3-rbd-2:17.0.0-11492.g9b3739e7.el8.x86_64 is already installed. 2022-04-10T01:15:40.451 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2022-04-10T01:15:40.452 INFO:teuthology.orchestra.run.smithi005.stdout:Nothing to do. 2022-04-10T01:15:40.452 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2022-04-10T01:15:40.512 INFO:teuthology.orchestra.run.smithi043.stdout:rbd-mirror-17.0.0-11492.g9b3739e7.el8.x86_64.rp 9.4 MB/s | 3.8 MB 00:00 2022-04-10T01:15:40.513 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2022-04-10T01:15:40.514 INFO:teuthology.orchestra.run.smithi043.stdout:Total 9.3 MB/s | 3.8 MB 00:00 2022-04-10T01:15:40.514 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2022-04-10T01:15:40.526 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2022-04-10T01:15:40.526 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2022-04-10T01:15:40.543 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install rbd-fuse 2022-04-10T01:15:40.594 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2022-04-10T01:15:40.597 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2022-04-10T01:15:41.107 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2022-04-10T01:15:41.134 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:02:26 ago on Sun 10 Apr 2022 01:13:15 AM UTC. 2022-04-10T01:15:41.342 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : rbd-mirror-2:17.0.0-11492.g9b3739e7.el8.x86_64 1/1 2022-04-10T01:15:41.715 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: rbd-mirror-2:17.0.0-11492.g9b3739e7.el8.x86_64 1/1 2022-04-10T01:15:42.044 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2022-04-10T01:15:42.045 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2022-04-10T01:15:42.045 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repo Size 2022-04-10T01:15:42.045 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2022-04-10T01:15:42.046 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2022-04-10T01:15:42.046 INFO:teuthology.orchestra.run.smithi005.stdout: rbd-fuse x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 93 k 2022-04-10T01:15:42.046 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:42.046 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2022-04-10T01:15:42.047 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2022-04-10T01:15:42.047 INFO:teuthology.orchestra.run.smithi005.stdout:Install 1 Package 2022-04-10T01:15:42.047 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:42.047 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 93 k 2022-04-10T01:15:42.048 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 280 k 2022-04-10T01:15:42.048 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2022-04-10T01:15:42.084 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : rbd-mirror-2:17.0.0-11492.g9b3739e7.el8.x86_64 1/1 2022-04-10T01:15:42.084 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:42.084 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2022-04-10T01:15:42.085 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-mirror-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:15:42.085 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:42.085 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2022-04-10T01:15:42.245 INFO:teuthology.orchestra.run.smithi005.stdout:rbd-fuse-17.0.0-11492.g9b3739e7.el8.x86_64.rpm 467 kB/s | 93 kB 00:00 2022-04-10T01:15:42.246 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2022-04-10T01:15:42.247 INFO:teuthology.orchestra.run.smithi005.stdout:Total 464 kB/s | 93 kB 00:00 2022-04-10T01:15:42.247 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2022-04-10T01:15:42.257 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2022-04-10T01:15:42.257 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2022-04-10T01:15:42.309 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install rbd-nbd 2022-04-10T01:15:42.310 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2022-04-10T01:15:42.313 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2022-04-10T01:15:42.423 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2022-04-10T01:15:42.665 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : rbd-fuse-2:17.0.0-11492.g9b3739e7.el8.x86_64 1/1 2022-04-10T01:15:42.905 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:02:27 ago on Sun 10 Apr 2022 01:13:15 AM UTC. 2022-04-10T01:15:42.924 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: rbd-fuse-2:17.0.0-11492.g9b3739e7.el8.x86_64 1/1 2022-04-10T01:15:43.329 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : rbd-fuse-2:17.0.0-11492.g9b3739e7.el8.x86_64 1/1 2022-04-10T01:15:43.329 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:43.330 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2022-04-10T01:15:43.330 INFO:teuthology.orchestra.run.smithi005.stdout: rbd-fuse-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:15:43.330 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:43.331 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2022-04-10T01:15:43.549 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install rbd-mirror 2022-04-10T01:15:43.824 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2022-04-10T01:15:43.825 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2022-04-10T01:15:43.825 INFO:teuthology.orchestra.run.smithi043.stdout: Package Architecture Version Repository Size 2022-04-10T01:15:43.826 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2022-04-10T01:15:43.826 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2022-04-10T01:15:43.827 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-nbd x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 175 k 2022-04-10T01:15:43.827 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:43.827 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2022-04-10T01:15:43.828 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2022-04-10T01:15:43.828 INFO:teuthology.orchestra.run.smithi043.stdout:Install 1 Package 2022-04-10T01:15:43.828 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:43.828 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 175 k 2022-04-10T01:15:43.829 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 567 k 2022-04-10T01:15:43.829 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2022-04-10T01:15:44.049 INFO:teuthology.orchestra.run.smithi043.stdout:rbd-nbd-17.0.0-11492.g9b3739e7.el8.x86_64.rpm 787 kB/s | 175 kB 00:00 2022-04-10T01:15:44.050 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2022-04-10T01:15:44.050 INFO:teuthology.orchestra.run.smithi043.stdout:Total 783 kB/s | 175 kB 00:00 2022-04-10T01:15:44.050 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2022-04-10T01:15:44.061 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2022-04-10T01:15:44.061 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2022-04-10T01:15:44.115 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2022-04-10T01:15:44.117 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2022-04-10T01:15:44.147 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:02:29 ago on Sun 10 Apr 2022 01:13:15 AM UTC. 2022-04-10T01:15:44.239 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2022-04-10T01:15:44.415 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : rbd-nbd-2:17.0.0-11492.g9b3739e7.el8.x86_64 1/1 2022-04-10T01:15:44.987 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: rbd-nbd-2:17.0.0-11492.g9b3739e7.el8.x86_64 1/1 2022-04-10T01:15:45.048 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2022-04-10T01:15:45.050 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2022-04-10T01:15:45.050 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repo Size 2022-04-10T01:15:45.050 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2022-04-10T01:15:45.050 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2022-04-10T01:15:45.051 INFO:teuthology.orchestra.run.smithi005.stdout: rbd-mirror x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 3.8 M 2022-04-10T01:15:45.051 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:45.051 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2022-04-10T01:15:45.051 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2022-04-10T01:15:45.052 INFO:teuthology.orchestra.run.smithi005.stdout:Install 1 Package 2022-04-10T01:15:45.052 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:45.052 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 3.8 M 2022-04-10T01:15:45.052 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 19 M 2022-04-10T01:15:45.052 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2022-04-10T01:15:45.361 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : rbd-nbd-2:17.0.0-11492.g9b3739e7.el8.x86_64 1/1 2022-04-10T01:15:45.361 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:45.361 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2022-04-10T01:15:45.361 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-nbd-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:15:45.362 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:45.362 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2022-04-10T01:15:45.571 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install rbd-mirror 2022-04-10T01:15:45.933 INFO:teuthology.orchestra.run.smithi005.stdout:rbd-mirror-17.0.0-11492.g9b3739e7.el8.x86_64.rp 4.3 MB/s | 3.8 MB 00:00 2022-04-10T01:15:45.935 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2022-04-10T01:15:45.935 INFO:teuthology.orchestra.run.smithi005.stdout:Total 4.3 MB/s | 3.8 MB 00:00 2022-04-10T01:15:45.935 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2022-04-10T01:15:45.947 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2022-04-10T01:15:45.947 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2022-04-10T01:15:46.015 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2022-04-10T01:15:46.018 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2022-04-10T01:15:46.165 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:02:31 ago on Sun 10 Apr 2022 01:13:15 AM UTC. 2022-04-10T01:15:46.534 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2022-04-10T01:15:46.766 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : rbd-mirror-2:17.0.0-11492.g9b3739e7.el8.x86_64 1/1 2022-04-10T01:15:46.997 INFO:teuthology.orchestra.run.smithi043.stdout:Package rbd-mirror-2:17.0.0-11492.g9b3739e7.el8.x86_64 is already installed. 2022-04-10T01:15:47.055 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2022-04-10T01:15:47.057 INFO:teuthology.orchestra.run.smithi043.stdout:Nothing to do. 2022-04-10T01:15:47.057 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2022-04-10T01:15:47.135 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: rbd-mirror-2:17.0.0-11492.g9b3739e7.el8.x86_64 1/1 2022-04-10T01:15:47.148 DEBUG:teuthology.parallel:result is None 2022-04-10T01:15:47.514 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : rbd-mirror-2:17.0.0-11492.g9b3739e7.el8.x86_64 1/1 2022-04-10T01:15:47.514 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:47.514 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2022-04-10T01:15:47.515 INFO:teuthology.orchestra.run.smithi005.stdout: rbd-mirror-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:15:47.515 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:47.515 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2022-04-10T01:15:47.729 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install rbd-nbd 2022-04-10T01:15:48.326 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:02:33 ago on Sun 10 Apr 2022 01:13:15 AM UTC. 2022-04-10T01:15:49.230 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2022-04-10T01:15:49.231 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2022-04-10T01:15:49.231 INFO:teuthology.orchestra.run.smithi005.stdout: Package Architecture Version Repository Size 2022-04-10T01:15:49.231 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2022-04-10T01:15:49.232 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2022-04-10T01:15:49.232 INFO:teuthology.orchestra.run.smithi005.stdout: rbd-nbd x86_64 2:17.0.0-11492.g9b3739e7.el8 ceph 175 k 2022-04-10T01:15:49.232 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:49.232 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2022-04-10T01:15:49.233 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2022-04-10T01:15:49.233 INFO:teuthology.orchestra.run.smithi005.stdout:Install 1 Package 2022-04-10T01:15:49.233 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:49.233 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 175 k 2022-04-10T01:15:49.233 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 567 k 2022-04-10T01:15:49.234 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2022-04-10T01:15:49.427 INFO:teuthology.orchestra.run.smithi005.stdout:rbd-nbd-17.0.0-11492.g9b3739e7.el8.x86_64.rpm 898 kB/s | 175 kB 00:00 2022-04-10T01:15:49.428 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2022-04-10T01:15:49.428 INFO:teuthology.orchestra.run.smithi005.stdout:Total 893 kB/s | 175 kB 00:00 2022-04-10T01:15:49.428 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2022-04-10T01:15:49.439 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2022-04-10T01:15:49.440 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2022-04-10T01:15:49.492 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2022-04-10T01:15:49.495 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2022-04-10T01:15:49.614 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2022-04-10T01:15:49.822 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : rbd-nbd-2:17.0.0-11492.g9b3739e7.el8.x86_64 1/1 2022-04-10T01:15:50.079 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: rbd-nbd-2:17.0.0-11492.g9b3739e7.el8.x86_64 1/1 2022-04-10T01:15:50.447 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : rbd-nbd-2:17.0.0-11492.g9b3739e7.el8.x86_64 1/1 2022-04-10T01:15:50.447 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:50.448 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2022-04-10T01:15:50.448 INFO:teuthology.orchestra.run.smithi005.stdout: rbd-nbd-2:17.0.0-11492.g9b3739e7.el8.x86_64 2022-04-10T01:15:50.448 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:50.448 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2022-04-10T01:15:50.723 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install rbd-mirror 2022-04-10T01:15:51.314 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:02:36 ago on Sun 10 Apr 2022 01:13:15 AM UTC. 2022-04-10T01:15:52.124 INFO:teuthology.orchestra.run.smithi005.stdout:Package rbd-mirror-2:17.0.0-11492.g9b3739e7.el8.x86_64 is already installed. 2022-04-10T01:15:52.185 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2022-04-10T01:15:52.186 INFO:teuthology.orchestra.run.smithi005.stdout:Nothing to do. 2022-04-10T01:15:52.187 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2022-04-10T01:15:52.264 DEBUG:teuthology.parallel:result is None 2022-04-10T01:15:52.265 DEBUG:teuthology.orchestra.run.smithi005:> sudo lsb_release -is 2022-04-10T01:15:52.301 INFO:teuthology.orchestra.run.smithi005.stdout:CentOSStream 2022-04-10T01:15:52.303 DEBUG:teuthology.misc:System to be installed: CentOSStream 2022-04-10T01:15:52.304 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=9b3739e792ef182b79445b926c7b7da461c05f52 2022-04-10T01:15:52.465 DEBUG:teuthology.orchestra.run.smithi005:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2022-04-10T01:15:52.488 INFO:teuthology.orchestra.run.smithi005.stdout:17.0.0-11492.g9b3739e7.el8 2022-04-10T01:15:52.488 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-11492.g9b3739e7.el8 2022-04-10T01:15:52.489 INFO:teuthology.task.install:The correct ceph version 17.0.0-11492.g9b3739e7 is installed. 2022-04-10T01:15:52.489 DEBUG:teuthology.orchestra.run.smithi043:> sudo lsb_release -is 2022-04-10T01:15:52.535 INFO:teuthology.orchestra.run.smithi043.stdout:CentOSStream 2022-04-10T01:15:52.536 DEBUG:teuthology.misc:System to be installed: CentOSStream 2022-04-10T01:15:52.537 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=9b3739e792ef182b79445b926c7b7da461c05f52 2022-04-10T01:15:52.686 DEBUG:teuthology.orchestra.run.smithi043:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2022-04-10T01:15:52.713 INFO:teuthology.orchestra.run.smithi043.stdout:17.0.0-11492.g9b3739e7.el8 2022-04-10T01:15:52.713 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-11492.g9b3739e7.el8 2022-04-10T01:15:52.714 INFO:teuthology.task.install:The correct ceph version 17.0.0-11492.g9b3739e7 is installed. 2022-04-10T01:15:52.714 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2022-04-10T01:15:52.715 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-04-10T01:15:52.715 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-04-10T01:15:52.743 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-04-10T01:15:52.744 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-04-10T01:15:52.784 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2022-04-10T01:15:52.785 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-04-10T01:15:52.785 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/usr/bin/daemon-helper 2022-04-10T01:15:52.814 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-04-10T01:15:52.881 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-04-10T01:15:52.882 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/usr/bin/daemon-helper 2022-04-10T01:15:52.913 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-04-10T01:15:52.984 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2022-04-10T01:15:52.985 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-04-10T01:15:52.985 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/usr/bin/adjust-ulimits 2022-04-10T01:15:53.014 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-04-10T01:15:53.081 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-04-10T01:15:53.082 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/usr/bin/adjust-ulimits 2022-04-10T01:15:53.113 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-04-10T01:15:53.183 INFO:teuthology.run_tasks:Running task ceph... 2022-04-10T01:15:53.262 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2022-04-10T01:15:53.263 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 777 /var/log/ceph 2022-04-10T01:15:53.265 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 777 /var/log/ceph 2022-04-10T01:15:53.295 INFO:tasks.ceph:Disabling ceph logrotate... 2022-04-10T01:15:53.296 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -f -- /etc/logrotate.d/ceph 2022-04-10T01:15:53.335 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -f -- /etc/logrotate.d/ceph 2022-04-10T01:15:53.365 INFO:tasks.ceph:Creating extra log directories... 2022-04-10T01:15:53.366 DEBUG:teuthology.orchestra.run.smithi005:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2022-04-10T01:15:53.404 DEBUG:teuthology.orchestra.run.smithi043:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2022-04-10T01:15:53.449 INFO:tasks.ceph:Creating ceph cluster cluster1... 2022-04-10T01:15:53.449 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2022-04-10T01:15:53.450 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/mrfreezeex-2022-04-09_15:53:21-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784449', 'branch': 'wip-mrfreezeex-multipeer', 'description': 'rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} msgr-failures/few objectstore/bluestore-hybrid policy/simple supported-random-distro$/{centos_8}}', 'email': 'arthur.outhenin-chalandre@cern.ch', 'first_in_suite': False, 'job_id': '6784449', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'mrfreezeex-2022-04-09_15:53:21-rbd-wip-mrfreezeex-multipeer-distro-default-smithi', 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-mrfreezeex-multipeer'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '9b3739e792ef182b79445b926c7b7da461c05f52'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '9b3739e792ef182b79445b926c7b7da461c05f52'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-mrfreezeex-multipeer', 'sha1': '9b3739e792ef182b79445b926c7b7da461c05f52'}}, 'owner': 'scheduled_mrfreezeex@teuthology', 'priority': 165, 'repo': 'git://git.ceph.com/ceph-ci.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'sha1': '9b3739e792ef182b79445b926c7b7da461c05f52', 'sleep_before_teardown': 0, 'suite': 'rbd', 'suite_branch': 'wip-mrfreezeex-multipeer', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_9b3739e792ef182b79445b926c7b7da461c05f52/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': '9b3739e792ef182b79445b926c7b7da461c05f52', 'targets': {'smithi005.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6a7IpOZPtZwIOOYNtsNKq1njcAjwzrK01ZhS9QRS+ifxFUZFe8h8edywJ1zqQAOZRNH+E+aKD6C1F+nyv2HdWrjIXqbyTlgyYxU/VS9RHc27G2rFDS9p7aogFlkCDXHKfx+BiYqnbGBmqtGKL0Os0Ur7m1uZ0hn34qmSiZS0XD4/wSTSwW6XwhkJCMxuMaoWzthqmojoClGBbutqSDhmOsLBOf3g+7PchS3HOPGYIUJ+rd9p2lkM+3YV76Lo/9Sj5TrYxONbGx8tZEnw+xkYwrGqTbxTAax8DX7Bws8jRXQeKekmgA1m+JzxmWB7vTIx2Nm9wo0iMLWQT53nbhx2Mcjg224BSYyYvQiJnPX46yv25nTVAzLnVXQ8lxTUXmacy2qwHRyOfe8BNYCisGmU5Z9suX/pFgnFEBbzf5wcFmW/KwsuTDa/A2nzRNRpsaWBv53ZA7jNXBXIYOYoaV3GFYLA2AdhvsGTMzpXYxa9f+QUxhJWFvLiDmBJk0kcwhCk=', 'smithi043.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxbBDJqyiaOu9Bbtq4sC6enMBXqfpHtMB96N/mf2lA/aaKGlTMNpp2Zye0gQ9z4HygetquzjtuApbZg+ldNp1ZyvL7lcYvBos5rNC2+zTYmhk6+9bvHmSv/51wk67aheDavAy1QvcABfu4F/qTSGfqoWAXv+BZXYH2KjS8M7Rrp9YbBjcoGRsGwwvAzHLqcK806WaVGWOJCdfcpsKWVrL3R/MbUZRVhvsi8PvAeGzMAnpsYO5nyNH2Syla720zcmld++unoK06P4fWkF8hFOq/ooREjY6oH+R5zVohi1pDrEAviod6TLCvAhauFyhBgT7bcCvLk2P3/VvC7CkiX17A4fLZYnMZpQHfgsSLvlbrLFgF767L2r+/ekL114Zhxc4EolKhuZlEodiLAvt3LVgb3FcGq+J68W/ejuEY9nmZEl4pfa/jsq5LVTFoBwjcpJd40J8QOcPMteeniG7rH77A+ScsxTVCvAhiSiLpwxNfZjU6FkwakxUJq+3gEjzy21s='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '9b3739e792ef182b79445b926c7b7da461c05f52'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '9b3739e792ef182b79445b926c7b7da461c05f52'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_IMAGE_MODE': 'snapshot', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology_branch': 'master', 'teuthology_sha1': 'a347b2ac5400a43ba76a678399b1deda7dae1a6c', 'timestamp': '2022-04-09_15:53:21', 'tube': 'smithi', 'user': 'mrfreezeex', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.22301'} 2022-04-10T01:15:53.451 DEBUG:teuthology.orchestra.run.smithi005:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2022-04-10T01:15:53.472 DEBUG:teuthology.orchestra.run.smithi043:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2022-04-10T01:15:53.492 DEBUG:teuthology.orchestra.run.smithi005:> sudo install -d -m0777 -- /var/run/ceph 2022-04-10T01:15:53.530 DEBUG:teuthology.orchestra.run.smithi043:> sudo install -d -m0777 -- /var/run/ceph 2022-04-10T01:15:53.560 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-04-10T01:15:53.561 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/scratch_devs of=/dev/stdout 2022-04-10T01:15:53.616 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-04-10T01:15:53.617 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-04-10T01:15:53.673 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-04-10T01:15:53.674 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-04-10T01:15:53.674 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 6h/6d Inode: 145323 Links: 1 2022-04-10T01:15:53.674 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-04-10T01:15:53.675 INFO:teuthology.orchestra.run.smithi005.stdout:Context: system_u:object_r:device_t:s0 2022-04-10T01:15:53.675 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2022-04-10 01:15:46.921939722 +0000 2022-04-10T01:15:53.675 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2022-04-10 01:12:37.900014856 +0000 2022-04-10T01:15:53.675 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2022-04-10 01:12:37.900014856 +0000 2022-04-10T01:15:53.676 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: - 2022-04-10T01:15:53.684 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2022-04-10T01:15:53.684 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2022-04-10T01:15:53.684 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 0.000117775 s, 4.3 MB/s 2022-04-10T01:15:53.687 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-04-10T01:15:53.744 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-04-10T01:15:53.745 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-04-10T01:15:53.745 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 6h/6d Inode: 153623 Links: 1 2022-04-10T01:15:53.745 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-04-10T01:15:53.745 INFO:teuthology.orchestra.run.smithi005.stdout:Context: system_u:object_r:device_t:s0 2022-04-10T01:15:53.746 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2022-04-10 01:15:46.921939722 +0000 2022-04-10T01:15:53.746 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2022-04-10 01:12:38.276007223 +0000 2022-04-10T01:15:53.746 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2022-04-10 01:12:38.276007223 +0000 2022-04-10T01:15:53.746 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: - 2022-04-10T01:15:53.754 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2022-04-10T01:15:53.755 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2022-04-10T01:15:53.755 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 0.000110429 s, 4.6 MB/s 2022-04-10T01:15:53.757 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-04-10T01:15:53.815 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-04-10T01:15:53.815 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-04-10T01:15:53.815 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 6h/6d Inode: 151010 Links: 1 2022-04-10T01:15:53.816 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-04-10T01:15:53.816 INFO:teuthology.orchestra.run.smithi005.stdout:Context: system_u:object_r:device_t:s0 2022-04-10T01:15:53.816 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2022-04-10 01:15:46.921939722 +0000 2022-04-10T01:15:53.816 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2022-04-10 01:12:38.603000586 +0000 2022-04-10T01:15:53.817 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2022-04-10 01:12:38.603000586 +0000 2022-04-10T01:15:53.817 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: - 2022-04-10T01:15:53.825 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2022-04-10T01:15:53.826 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2022-04-10T01:15:53.826 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 0.000100038 s, 5.1 MB/s 2022-04-10T01:15:53.828 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-04-10T01:15:53.886 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-04-10T01:15:53.887 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-04-10T01:15:53.887 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 6h/6d Inode: 152782 Links: 1 2022-04-10T01:15:53.887 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-04-10T01:15:53.887 INFO:teuthology.orchestra.run.smithi005.stdout:Context: system_u:object_r:device_t:s0 2022-04-10T01:15:53.888 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2022-04-10 01:15:46.921939722 +0000 2022-04-10T01:15:53.888 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2022-04-10 01:12:38.946993603 +0000 2022-04-10T01:15:53.888 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2022-04-10 01:12:38.946993603 +0000 2022-04-10T01:15:53.888 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: - 2022-04-10T01:15:53.896 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2022-04-10T01:15:53.896 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2022-04-10T01:15:53.897 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 0.000106697 s, 4.8 MB/s 2022-04-10T01:15:53.899 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2022-04-10T01:15:53.900 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi005.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2022-04-10T01:15:53.900 INFO:tasks.ceph:Generating config... 2022-04-10T01:15:53.901 INFO:tasks.ceph:[client] debug rbd = 20 2022-04-10T01:15:53.902 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2022-04-10T01:15:53.902 INFO:tasks.ceph:[client] log to stderr = False 2022-04-10T01:15:53.902 INFO:tasks.ceph:[client] rbd default features = 125 2022-04-10T01:15:53.903 INFO:tasks.ceph:[client] rbd mirror image policy type = simple 2022-04-10T01:15:53.903 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2022-04-10T01:15:53.903 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2022-04-10T01:15:53.904 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2022-04-10T01:15:53.904 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2022-04-10T01:15:53.904 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2022-04-10T01:15:53.905 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2022-04-10T01:15:53.905 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2022-04-10T01:15:53.905 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2022-04-10T01:15:53.906 INFO:tasks.ceph:[global] mon client directed command retry = 5 2022-04-10T01:15:53.906 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2022-04-10T01:15:53.906 INFO:tasks.ceph:[mgr] debug mgr = 20 2022-04-10T01:15:53.907 INFO:tasks.ceph:[mgr] debug ms = 1 2022-04-10T01:15:53.907 INFO:tasks.ceph:[mon] debug mon = 20 2022-04-10T01:15:53.907 INFO:tasks.ceph:[mon] debug ms = 1 2022-04-10T01:15:53.908 INFO:tasks.ceph:[mon] debug paxos = 20 2022-04-10T01:15:53.908 INFO:tasks.ceph:[osd] bluefs allocator = hybrid 2022-04-10T01:15:53.908 INFO:tasks.ceph:[osd] bluestore allocator = hybrid 2022-04-10T01:15:53.908 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2022-04-10T01:15:53.909 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2022-04-10T01:15:53.909 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2022-04-10T01:15:53.909 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2022-04-10T01:15:53.910 INFO:tasks.ceph:[osd] debug ms = 1 2022-04-10T01:15:53.910 INFO:tasks.ceph:[osd] debug osd = 20 2022-04-10T01:15:53.910 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2022-04-10T01:15:53.911 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2022-04-10T01:15:53.911 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2022-04-10T01:15:53.911 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2022-04-10T01:15:53.912 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2022-04-10T01:15:53.912 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2022-04-10T01:15:53.913 INFO:tasks.ceph:Setting up cluster1.mon.a... 2022-04-10T01:15:53.913 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2022-04-10T01:15:53.988 INFO:teuthology.orchestra.run.smithi005.stdout:creating /etc/ceph/cluster1.keyring 2022-04-10T01:15:53.991 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2022-04-10T01:15:54.033 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2022-04-10T01:15:54.101 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '172.21.15.5')] 2022-04-10T01:15:54.102 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.5', 'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2022-04-10T01:15:54.103 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-04-10T01:15:54.104 DEBUG:teuthology.orchestra.run.smithi005:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2022-04-10T01:15:54.158 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.5 --print /home/ubuntu/cephtest/cluster1.monmap 2022-04-10T01:15:54.237 INFO:teuthology.orchestra.run.smithi005.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2022-04-10T01:15:54.238 INFO:teuthology.orchestra.run.smithi005.stdout:monmaptool: generated fsid f7ffc1b1-9015-4973-aea5-caa6dd85ed1d 2022-04-10T01:15:54.238 INFO:teuthology.orchestra.run.smithi005.stdout:setting min_mon_release = octopus 2022-04-10T01:15:54.238 INFO:teuthology.orchestra.run.smithi005.stdout:epoch 0 2022-04-10T01:15:54.238 INFO:teuthology.orchestra.run.smithi005.stdout:fsid f7ffc1b1-9015-4973-aea5-caa6dd85ed1d 2022-04-10T01:15:54.239 INFO:teuthology.orchestra.run.smithi005.stdout:last_changed 2022-04-10T01:15:54.237336+0000 2022-04-10T01:15:54.239 INFO:teuthology.orchestra.run.smithi005.stdout:created 2022-04-10T01:15:54.237336+0000 2022-04-10T01:15:54.239 INFO:teuthology.orchestra.run.smithi005.stdout:min_mon_release 15 (octopus) 2022-04-10T01:15:54.239 INFO:teuthology.orchestra.run.smithi005.stdout:election_strategy: 1 2022-04-10T01:15:54.240 INFO:teuthology.orchestra.run.smithi005.stdout:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2022-04-10T01:15:54.240 INFO:teuthology.orchestra.run.smithi005.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2022-04-10T01:15:54.242 DEBUG:teuthology.orchestra.run.smithi005:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2022-04-10T01:15:54.299 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID f7ffc1b1-9015-4973-aea5-caa6dd85ed1d... 2022-04-10T01:15:54.300 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2022-04-10T01:15:54.341 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2022-04-10T01:15:54.385 INFO:teuthology.orchestra.run.smithi005.stdout:[global] 2022-04-10T01:15:54.385 INFO:teuthology.orchestra.run.smithi005.stdout: chdir = "" 2022-04-10T01:15:54.385 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-04-10T01:15:54.386 INFO:teuthology.orchestra.run.smithi005.stdout: auth supported = cephx 2022-04-10T01:15:54.386 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:54.386 INFO:teuthology.orchestra.run.smithi005.stdout: filestore xattr use omap = true 2022-04-10T01:15:54.386 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:54.387 INFO:teuthology.orchestra.run.smithi005.stdout: mon clock drift allowed = 1.000 2022-04-10T01:15:54.387 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:54.387 INFO:teuthology.orchestra.run.smithi005.stdout: osd crush chooseleaf type = 0 2022-04-10T01:15:54.387 INFO:teuthology.orchestra.run.smithi005.stdout: auth debug = true 2022-04-10T01:15:54.387 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:54.388 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on old message = true 2022-04-10T01:15:54.388 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on bug = true 2022-04-10T01:15:54.388 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:54.388 INFO:teuthology.orchestra.run.smithi005.stdout: mon max pg per osd = 10000 # >= luminous 2022-04-10T01:15:54.388 INFO:teuthology.orchestra.run.smithi005.stdout: mon pg warn max object skew = 0 2022-04-10T01:15:54.389 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:54.389 INFO:teuthology.orchestra.run.smithi005.stdout: # disable pg_autoscaler by default for new pools 2022-04-10T01:15:54.389 INFO:teuthology.orchestra.run.smithi005.stdout: osd_pool_default_pg_autoscale_mode = off 2022-04-10T01:15:54.389 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:54.390 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default size = 2 2022-04-10T01:15:54.390 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:54.390 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow primary affinity = true 2022-04-10T01:15:54.390 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow pg remap = true 2022-04-10T01:15:54.390 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on legacy crush tunables = false 2022-04-10T01:15:54.391 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on crush straw calc version zero = false 2022-04-10T01:15:54.391 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on no sortbitwise = false 2022-04-10T01:15:54.391 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on osd down out interval zero = false 2022-04-10T01:15:54.391 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on too few osds = false 2022-04-10T01:15:54.392 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-04-10T01:15:54.392 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_no_redundancy = false 2022-04-10T01:15:54.392 INFO:teuthology.orchestra.run.smithi005.stdout: mon_allow_pool_size_one = true 2022-04-10T01:15:54.392 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:54.392 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2022-04-10T01:15:54.393 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:54.393 INFO:teuthology.orchestra.run.smithi005.stdout: osd default data pool replay window = 5 2022-04-10T01:15:54.393 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:54.393 INFO:teuthology.orchestra.run.smithi005.stdout: mon allow pool delete = true 2022-04-10T01:15:54.394 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:54.394 INFO:teuthology.orchestra.run.smithi005.stdout: mon cluster log file level = debug 2022-04-10T01:15:54.394 INFO:teuthology.orchestra.run.smithi005.stdout: debug asserts on shutdown = true 2022-04-10T01:15:54.394 INFO:teuthology.orchestra.run.smithi005.stdout: mon health detail to clog = false 2022-04-10T01:15:54.394 INFO:teuthology.orchestra.run.smithi005.stdout: mon host = 172.21.15.5 2022-04-10T01:15:54.395 INFO:teuthology.orchestra.run.smithi005.stdout: mon client directed command retry = 5 2022-04-10T01:15:54.395 INFO:teuthology.orchestra.run.smithi005.stdout: ms inject socket failures = 5000 2022-04-10T01:15:54.395 INFO:teuthology.orchestra.run.smithi005.stdout: fsid = f7ffc1b1-9015-4973-aea5-caa6dd85ed1d 2022-04-10T01:15:54.395 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:54.396 INFO:teuthology.orchestra.run.smithi005.stdout:[osd] 2022-04-10T01:15:54.396 INFO:teuthology.orchestra.run.smithi005.stdout: osd journal size = 100 2022-04-10T01:15:54.396 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:54.396 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub load threshold = 5.0 2022-04-10T01:15:54.396 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub max interval = 600 2022-04-10T01:15:54.397 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:54.397 INFO:teuthology.orchestra.run.smithi005.stdout: osd recover clone overlap = true 2022-04-10T01:15:54.397 INFO:teuthology.orchestra.run.smithi005.stdout: osd recovery max chunk = 1048576 2022-04-10T01:15:54.397 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:54.397 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug shutdown = true 2022-04-10T01:15:54.398 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug op order = true 2022-04-10T01:15:54.398 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug verify stray on activate = true 2022-04-10T01:15:54.398 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:54.399 INFO:teuthology.orchestra.run.smithi005.stdout: osd open classes on start = true 2022-04-10T01:15:54.399 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug pg log writeout = true 2022-04-10T01:15:54.399 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:54.399 INFO:teuthology.orchestra.run.smithi005.stdout: osd deep scrub update digest min age = 30 2022-04-10T01:15:54.399 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:54.400 INFO:teuthology.orchestra.run.smithi005.stdout: osd map max advance = 10 2022-04-10T01:15:54.400 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:54.400 INFO:teuthology.orchestra.run.smithi005.stdout: journal zero on create = true 2022-04-10T01:15:54.400 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:54.401 INFO:teuthology.orchestra.run.smithi005.stdout: filestore ondisk finisher threads = 3 2022-04-10T01:15:54.401 INFO:teuthology.orchestra.run.smithi005.stdout: filestore apply finisher threads = 3 2022-04-10T01:15:54.401 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:54.401 INFO:teuthology.orchestra.run.smithi005.stdout: bdev debug aio = true 2022-04-10T01:15:54.401 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug misdirected ops = true 2022-04-10T01:15:54.402 INFO:teuthology.orchestra.run.smithi005.stdout: bluefs allocator = hybrid 2022-04-10T01:15:54.402 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore allocator = hybrid 2022-04-10T01:15:54.402 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore block size = 96636764160 2022-04-10T01:15:54.402 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore fsck on mount = True 2022-04-10T01:15:54.403 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluefs = 1/20 2022-04-10T01:15:54.403 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluestore = 1/20 2022-04-10T01:15:54.403 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2022-04-10T01:15:54.403 INFO:teuthology.orchestra.run.smithi005.stdout: debug osd = 20 2022-04-10T01:15:54.403 INFO:teuthology.orchestra.run.smithi005.stdout: debug rocksdb = 4/10 2022-04-10T01:15:54.404 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd backfillfull_ratio = 0.85 2022-04-10T01:15:54.404 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd full ratio = 0.9 2022-04-10T01:15:54.404 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd nearfull ratio = 0.8 2022-04-10T01:15:54.404 INFO:teuthology.orchestra.run.smithi005.stdout: osd failsafe full ratio = 0.95 2022-04-10T01:15:54.405 INFO:teuthology.orchestra.run.smithi005.stdout: osd objectstore = bluestore 2022-04-10T01:15:54.405 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:54.405 INFO:teuthology.orchestra.run.smithi005.stdout:[mgr] 2022-04-10T01:15:54.405 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2022-04-10T01:15:54.405 INFO:teuthology.orchestra.run.smithi005.stdout: debug mgr = 20 2022-04-10T01:15:54.406 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2022-04-10T01:15:54.406 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2022-04-10T01:15:54.406 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2022-04-10T01:15:54.406 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2022-04-10T01:15:54.406 INFO:teuthology.orchestra.run.smithi005.stdout: mgr/telemetry/nag = false 2022-04-10T01:15:54.407 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:54.407 INFO:teuthology.orchestra.run.smithi005.stdout:[mon] 2022-04-10T01:15:54.407 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2022-04-10T01:15:54.407 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2022-04-10T01:15:54.407 INFO:teuthology.orchestra.run.smithi005.stdout: debug paxos = 20 2022-04-10T01:15:54.408 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2022-04-10T01:15:54.408 INFO:teuthology.orchestra.run.smithi005.stdout: mon data avail warn = 5 2022-04-10T01:15:54.408 INFO:teuthology.orchestra.run.smithi005.stdout: mon mgr mkfs grace = 240 2022-04-10T01:15:54.408 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2022-04-10T01:15:54.408 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd reporter subtree level = osd 2022-04-10T01:15:54.409 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd prime pg temp = true 2022-04-10T01:15:54.409 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2022-04-10T01:15:54.409 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:54.409 INFO:teuthology.orchestra.run.smithi005.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-04-10T01:15:54.410 INFO:teuthology.orchestra.run.smithi005.stdout: auth mon ticket ttl = 660 # 11m 2022-04-10T01:15:54.410 INFO:teuthology.orchestra.run.smithi005.stdout: auth service ticket ttl = 240 # 4m 2022-04-10T01:15:54.410 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:54.410 INFO:teuthology.orchestra.run.smithi005.stdout: # don't complain about insecure global_id in the test suite 2022-04-10T01:15:54.410 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-04-10T01:15:54.411 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-04-10T01:15:54.411 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:54.411 INFO:teuthology.orchestra.run.smithi005.stdout: # 1m isn't quite enough 2022-04-10T01:15:54.411 INFO:teuthology.orchestra.run.smithi005.stdout: mon_down_mkfs_grace = 2m 2022-04-10T01:15:54.411 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:54.412 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_filestore_osds = false 2022-04-10T01:15:54.412 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:15:54.412 INFO:teuthology.orchestra.run.smithi005.stdout:[client] 2022-04-10T01:15:54.412 INFO:teuthology.orchestra.run.smithi005.stdout: rgw cache enabled = true 2022-04-10T01:15:54.412 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable ops log = true 2022-04-10T01:15:54.413 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable usage log = true 2022-04-10T01:15:54.413 INFO:teuthology.orchestra.run.smithi005.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-04-10T01:15:54.413 INFO:teuthology.orchestra.run.smithi005.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-04-10T01:15:54.413 INFO:teuthology.orchestra.run.smithi005.stdout: debug rbd = 20 2022-04-10T01:15:54.413 INFO:teuthology.orchestra.run.smithi005.stdout: debug rbd_mirror = 15 2022-04-10T01:15:54.414 INFO:teuthology.orchestra.run.smithi005.stdout: log to stderr = False 2022-04-10T01:15:54.414 INFO:teuthology.orchestra.run.smithi005.stdout: rbd default features = 125 2022-04-10T01:15:54.414 INFO:teuthology.orchestra.run.smithi005.stdout: rbd mirror image policy type = simple 2022-04-10T01:15:54.414 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.a] 2022-04-10T01:15:54.415 INFO:teuthology.orchestra.run.smithi005.stdout:[client.mirror.0] 2022-04-10T01:15:54.415 INFO:teuthology.orchestra.run.smithi005.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2022-04-10T01:15:54.415 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2022-04-10T01:15:54.415 INFO:teuthology.orchestra.run.smithi005.stdout:[client.mirror.1] 2022-04-10T01:15:54.415 INFO:teuthology.orchestra.run.smithi005.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2022-04-10T01:15:54.416 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2022-04-10T01:15:54.416 INFO:teuthology.orchestra.run.smithi005.stdout:[client.mirror.2] 2022-04-10T01:15:54.416 INFO:teuthology.orchestra.run.smithi005.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2022-04-10T01:15:54.416 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2022-04-10T01:15:54.417 INFO:teuthology.orchestra.run.smithi005.stdout:[client.mirror.3] 2022-04-10T01:15:54.417 INFO:teuthology.orchestra.run.smithi005.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2022-04-10T01:15:54.417 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2022-04-10T01:15:54.419 INFO:teuthology.orchestra.run.smithi043.stdout:[global] 2022-04-10T01:15:54.419 INFO:teuthology.orchestra.run.smithi043.stdout: chdir = "" 2022-04-10T01:15:54.419 INFO:teuthology.orchestra.run.smithi043.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-04-10T01:15:54.420 INFO:teuthology.orchestra.run.smithi043.stdout: auth supported = cephx 2022-04-10T01:15:54.420 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:54.420 INFO:teuthology.orchestra.run.smithi043.stdout: filestore xattr use omap = true 2022-04-10T01:15:54.420 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:54.421 INFO:teuthology.orchestra.run.smithi043.stdout: mon clock drift allowed = 1.000 2022-04-10T01:15:54.421 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:54.421 INFO:teuthology.orchestra.run.smithi043.stdout: osd crush chooseleaf type = 0 2022-04-10T01:15:54.421 INFO:teuthology.orchestra.run.smithi043.stdout: auth debug = true 2022-04-10T01:15:54.421 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:54.422 INFO:teuthology.orchestra.run.smithi043.stdout: ms die on old message = true 2022-04-10T01:15:54.422 INFO:teuthology.orchestra.run.smithi043.stdout: ms die on bug = true 2022-04-10T01:15:54.422 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:54.422 INFO:teuthology.orchestra.run.smithi043.stdout: mon max pg per osd = 10000 # >= luminous 2022-04-10T01:15:54.422 INFO:teuthology.orchestra.run.smithi043.stdout: mon pg warn max object skew = 0 2022-04-10T01:15:54.423 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:54.423 INFO:teuthology.orchestra.run.smithi043.stdout: # disable pg_autoscaler by default for new pools 2022-04-10T01:15:54.423 INFO:teuthology.orchestra.run.smithi043.stdout: osd_pool_default_pg_autoscale_mode = off 2022-04-10T01:15:54.423 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:54.424 INFO:teuthology.orchestra.run.smithi043.stdout: osd pool default size = 2 2022-04-10T01:15:54.424 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:54.424 INFO:teuthology.orchestra.run.smithi043.stdout: mon osd allow primary affinity = true 2022-04-10T01:15:54.424 INFO:teuthology.orchestra.run.smithi043.stdout: mon osd allow pg remap = true 2022-04-10T01:15:54.424 INFO:teuthology.orchestra.run.smithi043.stdout: mon warn on legacy crush tunables = false 2022-04-10T01:15:54.425 INFO:teuthology.orchestra.run.smithi043.stdout: mon warn on crush straw calc version zero = false 2022-04-10T01:15:54.425 INFO:teuthology.orchestra.run.smithi043.stdout: mon warn on no sortbitwise = false 2022-04-10T01:15:54.425 INFO:teuthology.orchestra.run.smithi043.stdout: mon warn on osd down out interval zero = false 2022-04-10T01:15:54.425 INFO:teuthology.orchestra.run.smithi043.stdout: mon warn on too few osds = false 2022-04-10T01:15:54.425 INFO:teuthology.orchestra.run.smithi043.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-04-10T01:15:54.426 INFO:teuthology.orchestra.run.smithi043.stdout: mon_warn_on_pool_no_redundancy = false 2022-04-10T01:15:54.426 INFO:teuthology.orchestra.run.smithi043.stdout: mon_allow_pool_size_one = true 2022-04-10T01:15:54.426 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:54.426 INFO:teuthology.orchestra.run.smithi043.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2022-04-10T01:15:54.427 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:54.427 INFO:teuthology.orchestra.run.smithi043.stdout: osd default data pool replay window = 5 2022-04-10T01:15:54.427 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:54.428 INFO:teuthology.orchestra.run.smithi043.stdout: mon allow pool delete = true 2022-04-10T01:15:54.428 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:54.428 INFO:teuthology.orchestra.run.smithi043.stdout: mon cluster log file level = debug 2022-04-10T01:15:54.428 INFO:teuthology.orchestra.run.smithi043.stdout: debug asserts on shutdown = true 2022-04-10T01:15:54.428 INFO:teuthology.orchestra.run.smithi043.stdout: mon health detail to clog = false 2022-04-10T01:15:54.429 INFO:teuthology.orchestra.run.smithi043.stdout: mon host = 172.21.15.5 2022-04-10T01:15:54.429 INFO:teuthology.orchestra.run.smithi043.stdout: mon client directed command retry = 5 2022-04-10T01:15:54.429 INFO:teuthology.orchestra.run.smithi043.stdout: ms inject socket failures = 5000 2022-04-10T01:15:54.429 INFO:teuthology.orchestra.run.smithi043.stdout: fsid = f7ffc1b1-9015-4973-aea5-caa6dd85ed1d 2022-04-10T01:15:54.430 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:54.430 INFO:teuthology.orchestra.run.smithi043.stdout:[osd] 2022-04-10T01:15:54.430 INFO:teuthology.orchestra.run.smithi043.stdout: osd journal size = 100 2022-04-10T01:15:54.430 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:54.430 INFO:teuthology.orchestra.run.smithi043.stdout: osd scrub load threshold = 5.0 2022-04-10T01:15:54.431 INFO:teuthology.orchestra.run.smithi043.stdout: osd scrub max interval = 600 2022-04-10T01:15:54.431 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:54.431 INFO:teuthology.orchestra.run.smithi043.stdout: osd recover clone overlap = true 2022-04-10T01:15:54.431 INFO:teuthology.orchestra.run.smithi043.stdout: osd recovery max chunk = 1048576 2022-04-10T01:15:54.432 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:54.432 INFO:teuthology.orchestra.run.smithi043.stdout: osd debug shutdown = true 2022-04-10T01:15:54.432 INFO:teuthology.orchestra.run.smithi043.stdout: osd debug op order = true 2022-04-10T01:15:54.432 INFO:teuthology.orchestra.run.smithi043.stdout: osd debug verify stray on activate = true 2022-04-10T01:15:54.432 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:54.433 INFO:teuthology.orchestra.run.smithi043.stdout: osd open classes on start = true 2022-04-10T01:15:54.433 INFO:teuthology.orchestra.run.smithi043.stdout: osd debug pg log writeout = true 2022-04-10T01:15:54.433 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:54.433 INFO:teuthology.orchestra.run.smithi043.stdout: osd deep scrub update digest min age = 30 2022-04-10T01:15:54.433 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:54.434 INFO:teuthology.orchestra.run.smithi043.stdout: osd map max advance = 10 2022-04-10T01:15:54.434 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:54.434 INFO:teuthology.orchestra.run.smithi043.stdout: journal zero on create = true 2022-04-10T01:15:54.434 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:54.435 INFO:teuthology.orchestra.run.smithi043.stdout: filestore ondisk finisher threads = 3 2022-04-10T01:15:54.435 INFO:teuthology.orchestra.run.smithi043.stdout: filestore apply finisher threads = 3 2022-04-10T01:15:54.435 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:54.435 INFO:teuthology.orchestra.run.smithi043.stdout: bdev debug aio = true 2022-04-10T01:15:54.435 INFO:teuthology.orchestra.run.smithi043.stdout: osd debug misdirected ops = true 2022-04-10T01:15:54.436 INFO:teuthology.orchestra.run.smithi043.stdout: bluefs allocator = hybrid 2022-04-10T01:15:54.436 INFO:teuthology.orchestra.run.smithi043.stdout: bluestore allocator = hybrid 2022-04-10T01:15:54.436 INFO:teuthology.orchestra.run.smithi043.stdout: bluestore block size = 96636764160 2022-04-10T01:15:54.436 INFO:teuthology.orchestra.run.smithi043.stdout: bluestore fsck on mount = True 2022-04-10T01:15:54.436 INFO:teuthology.orchestra.run.smithi043.stdout: debug bluefs = 1/20 2022-04-10T01:15:54.437 INFO:teuthology.orchestra.run.smithi043.stdout: debug bluestore = 1/20 2022-04-10T01:15:54.437 INFO:teuthology.orchestra.run.smithi043.stdout: debug ms = 1 2022-04-10T01:15:54.437 INFO:teuthology.orchestra.run.smithi043.stdout: debug osd = 20 2022-04-10T01:15:54.437 INFO:teuthology.orchestra.run.smithi043.stdout: debug rocksdb = 4/10 2022-04-10T01:15:54.438 INFO:teuthology.orchestra.run.smithi043.stdout: mon osd backfillfull_ratio = 0.85 2022-04-10T01:15:54.438 INFO:teuthology.orchestra.run.smithi043.stdout: mon osd full ratio = 0.9 2022-04-10T01:15:54.438 INFO:teuthology.orchestra.run.smithi043.stdout: mon osd nearfull ratio = 0.8 2022-04-10T01:15:54.438 INFO:teuthology.orchestra.run.smithi043.stdout: osd failsafe full ratio = 0.95 2022-04-10T01:15:54.438 INFO:teuthology.orchestra.run.smithi043.stdout: osd objectstore = bluestore 2022-04-10T01:15:54.439 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:54.439 INFO:teuthology.orchestra.run.smithi043.stdout:[mgr] 2022-04-10T01:15:54.439 INFO:teuthology.orchestra.run.smithi043.stdout: debug ms = 1 2022-04-10T01:15:54.439 INFO:teuthology.orchestra.run.smithi043.stdout: debug mgr = 20 2022-04-10T01:15:54.439 INFO:teuthology.orchestra.run.smithi043.stdout: debug mon = 20 2022-04-10T01:15:54.440 INFO:teuthology.orchestra.run.smithi043.stdout: debug auth = 20 2022-04-10T01:15:54.440 INFO:teuthology.orchestra.run.smithi043.stdout: mon reweight min pgs per osd = 4 2022-04-10T01:15:54.440 INFO:teuthology.orchestra.run.smithi043.stdout: mon reweight min bytes per osd = 10 2022-04-10T01:15:54.440 INFO:teuthology.orchestra.run.smithi043.stdout: mgr/telemetry/nag = false 2022-04-10T01:15:54.441 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:54.441 INFO:teuthology.orchestra.run.smithi043.stdout:[mon] 2022-04-10T01:15:54.441 INFO:teuthology.orchestra.run.smithi043.stdout: debug ms = 1 2022-04-10T01:15:54.441 INFO:teuthology.orchestra.run.smithi043.stdout: debug mon = 20 2022-04-10T01:15:54.441 INFO:teuthology.orchestra.run.smithi043.stdout: debug paxos = 20 2022-04-10T01:15:54.442 INFO:teuthology.orchestra.run.smithi043.stdout: debug auth = 20 2022-04-10T01:15:54.442 INFO:teuthology.orchestra.run.smithi043.stdout: mon data avail warn = 5 2022-04-10T01:15:54.442 INFO:teuthology.orchestra.run.smithi043.stdout: mon mgr mkfs grace = 240 2022-04-10T01:15:54.442 INFO:teuthology.orchestra.run.smithi043.stdout: mon reweight min pgs per osd = 4 2022-04-10T01:15:54.443 INFO:teuthology.orchestra.run.smithi043.stdout: mon osd reporter subtree level = osd 2022-04-10T01:15:54.443 INFO:teuthology.orchestra.run.smithi043.stdout: mon osd prime pg temp = true 2022-04-10T01:15:54.443 INFO:teuthology.orchestra.run.smithi043.stdout: mon reweight min bytes per osd = 10 2022-04-10T01:15:54.443 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:54.443 INFO:teuthology.orchestra.run.smithi043.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-04-10T01:15:54.444 INFO:teuthology.orchestra.run.smithi043.stdout: auth mon ticket ttl = 660 # 11m 2022-04-10T01:15:54.444 INFO:teuthology.orchestra.run.smithi043.stdout: auth service ticket ttl = 240 # 4m 2022-04-10T01:15:54.444 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:54.444 INFO:teuthology.orchestra.run.smithi043.stdout: # don't complain about insecure global_id in the test suite 2022-04-10T01:15:54.445 INFO:teuthology.orchestra.run.smithi043.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-04-10T01:15:54.445 INFO:teuthology.orchestra.run.smithi043.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-04-10T01:15:54.445 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:54.445 INFO:teuthology.orchestra.run.smithi043.stdout: # 1m isn't quite enough 2022-04-10T01:15:54.445 INFO:teuthology.orchestra.run.smithi043.stdout: mon_down_mkfs_grace = 2m 2022-04-10T01:15:54.446 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:54.446 INFO:teuthology.orchestra.run.smithi043.stdout: mon_warn_on_filestore_osds = false 2022-04-10T01:15:54.446 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:15:54.446 INFO:teuthology.orchestra.run.smithi043.stdout:[client] 2022-04-10T01:15:54.446 INFO:teuthology.orchestra.run.smithi043.stdout: rgw cache enabled = true 2022-04-10T01:15:54.447 INFO:teuthology.orchestra.run.smithi043.stdout: rgw enable ops log = true 2022-04-10T01:15:54.447 INFO:teuthology.orchestra.run.smithi043.stdout: rgw enable usage log = true 2022-04-10T01:15:54.447 INFO:teuthology.orchestra.run.smithi043.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-04-10T01:15:54.447 INFO:teuthology.orchestra.run.smithi043.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-04-10T01:15:54.447 INFO:teuthology.orchestra.run.smithi043.stdout: debug rbd = 20 2022-04-10T01:15:54.448 INFO:teuthology.orchestra.run.smithi043.stdout: debug rbd_mirror = 15 2022-04-10T01:15:54.448 INFO:teuthology.orchestra.run.smithi043.stdout: log to stderr = False 2022-04-10T01:15:54.448 INFO:teuthology.orchestra.run.smithi043.stdout: rbd default features = 125 2022-04-10T01:15:54.448 INFO:teuthology.orchestra.run.smithi043.stdout: rbd mirror image policy type = simple 2022-04-10T01:15:54.449 INFO:teuthology.orchestra.run.smithi043.stdout:[mon.a] 2022-04-10T01:15:54.449 INFO:teuthology.orchestra.run.smithi043.stdout:[client.mirror.0] 2022-04-10T01:15:54.449 INFO:teuthology.orchestra.run.smithi043.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2022-04-10T01:15:54.449 INFO:teuthology.orchestra.run.smithi043.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2022-04-10T01:15:54.449 INFO:teuthology.orchestra.run.smithi043.stdout:[client.mirror.1] 2022-04-10T01:15:54.450 INFO:teuthology.orchestra.run.smithi043.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2022-04-10T01:15:54.450 INFO:teuthology.orchestra.run.smithi043.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2022-04-10T01:15:54.450 INFO:teuthology.orchestra.run.smithi043.stdout:[client.mirror.2] 2022-04-10T01:15:54.450 INFO:teuthology.orchestra.run.smithi043.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2022-04-10T01:15:54.450 INFO:teuthology.orchestra.run.smithi043.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2022-04-10T01:15:54.451 INFO:teuthology.orchestra.run.smithi043.stdout:[client.mirror.3] 2022-04-10T01:15:54.451 INFO:teuthology.orchestra.run.smithi043.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2022-04-10T01:15:54.451 INFO:teuthology.orchestra.run.smithi043.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2022-04-10T01:15:54.452 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2022-04-10T01:15:54.452 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2022-04-10T01:15:54.512 INFO:tasks.ceph:Copying monmap to all nodes... 2022-04-10T01:15:54.513 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-04-10T01:15:54.513 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2022-04-10T01:15:54.530 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-04-10T01:15:54.531 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2022-04-10T01:15:54.587 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi005.front.sepia.ceph.com 2022-04-10T01:15:54.588 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-04-10T01:15:54.588 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/ceph/cluster1.keyring 2022-04-10T01:15:54.588 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2022-04-10T01:15:54.664 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-04-10T01:15:54.665 DEBUG:teuthology.orchestra.run.smithi005:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2022-04-10T01:15:54.723 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi043.front.sepia.ceph.com 2022-04-10T01:15:54.724 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-04-10T01:15:54.724 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/ceph/cluster1.keyring 2022-04-10T01:15:54.724 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2022-04-10T01:15:54.769 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-04-10T01:15:54.770 DEBUG:teuthology.orchestra.run.smithi043:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2022-04-10T01:15:54.829 INFO:tasks.ceph:Setting up mon nodes... 2022-04-10T01:15:54.830 INFO:tasks.ceph:Setting up mgr nodes... 2022-04-10T01:15:54.830 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2022-04-10T01:15:54.882 INFO:teuthology.orchestra.run.smithi005.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2022-04-10T01:15:54.884 INFO:tasks.ceph:Setting up mds nodes... 2022-04-10T01:15:54.885 INFO:tasks.ceph_client:Setting up client nodes... 2022-04-10T01:15:54.885 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2022-04-10T01:15:54.926 INFO:teuthology.orchestra.run.smithi005.stdout:creating /etc/ceph/cluster1.client.0.keyring 2022-04-10T01:15:54.939 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2022-04-10T01:15:55.022 INFO:teuthology.orchestra.run.smithi043.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2022-04-10T01:15:55.036 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2022-04-10T01:15:55.080 INFO:teuthology.orchestra.run.smithi043.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2022-04-10T01:15:55.094 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2022-04-10T01:15:55.176 INFO:teuthology.orchestra.run.smithi043.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2022-04-10T01:15:55.189 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2022-04-10T01:15:55.270 INFO:teuthology.orchestra.run.smithi043.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2022-04-10T01:15:55.283 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2022-04-10T01:15:55.364 INFO:teuthology.orchestra.run.smithi043.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2022-04-10T01:15:55.377 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2022-04-10T01:15:55.458 INFO:teuthology.orchestra.run.smithi043.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2022-04-10T01:15:55.471 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2022-04-10T01:15:55.552 INFO:teuthology.orchestra.run.smithi043.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2022-04-10T01:15:55.565 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2022-04-10T01:15:55.646 INFO:teuthology.orchestra.run.smithi043.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2022-04-10T01:15:55.659 INFO:tasks.ceph:Running mkfs on osd nodes... 2022-04-10T01:15:55.659 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi005.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2022-04-10T01:15:55.660 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2022-04-10T01:15:55.693 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2022-04-10T01:15:55.694 INFO:tasks.ceph:role: cluster1.osd.0 2022-04-10T01:15:55.694 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi005.front.sepia.ceph.com 2022-04-10T01:15:55.694 DEBUG:teuthology.orchestra.run.smithi005:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2022-04-10T01:15:55.765 INFO:teuthology.orchestra.run.smithi005.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2022-04-10T01:15:55.766 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 attr=2, projid32bit=1 2022-04-10T01:15:55.766 INFO:teuthology.orchestra.run.smithi005.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-04-10T01:15:55.767 INFO:teuthology.orchestra.run.smithi005.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-04-10T01:15:55.767 INFO:teuthology.orchestra.run.smithi005.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-04-10T01:15:55.768 INFO:teuthology.orchestra.run.smithi005.stdout: = sunit=0 swidth=0 blks 2022-04-10T01:15:55.768 INFO:teuthology.orchestra.run.smithi005.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-04-10T01:15:55.769 INFO:teuthology.orchestra.run.smithi005.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-04-10T01:15:55.769 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-04-10T01:15:55.769 INFO:teuthology.orchestra.run.smithi005.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-04-10T01:15:55.788 INFO:teuthology.orchestra.run.smithi005.stdout:Discarding blocks...Done. 2022-04-10T01:15:55.790 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi005.front.sepia.ceph.com -o noatime 2022-04-10T01:15:55.791 DEBUG:teuthology.orchestra.run.smithi005:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2022-04-10T01:15:55.874 DEBUG:teuthology.orchestra.run.smithi005:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2022-04-10T01:15:55.948 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2022-04-10T01:15:56.018 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2022-04-10T01:15:56.018 INFO:tasks.ceph:role: cluster1.osd.1 2022-04-10T01:15:56.019 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi005.front.sepia.ceph.com 2022-04-10T01:15:56.019 DEBUG:teuthology.orchestra.run.smithi005:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2022-04-10T01:15:56.085 INFO:teuthology.orchestra.run.smithi005.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2022-04-10T01:15:56.086 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 attr=2, projid32bit=1 2022-04-10T01:15:56.086 INFO:teuthology.orchestra.run.smithi005.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-04-10T01:15:56.086 INFO:teuthology.orchestra.run.smithi005.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-04-10T01:15:56.087 INFO:teuthology.orchestra.run.smithi005.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-04-10T01:15:56.087 INFO:teuthology.orchestra.run.smithi005.stdout: = sunit=0 swidth=0 blks 2022-04-10T01:15:56.087 INFO:teuthology.orchestra.run.smithi005.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-04-10T01:15:56.088 INFO:teuthology.orchestra.run.smithi005.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-04-10T01:15:56.088 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-04-10T01:15:56.088 INFO:teuthology.orchestra.run.smithi005.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-04-10T01:15:56.108 INFO:teuthology.orchestra.run.smithi005.stdout:Discarding blocks...Done. 2022-04-10T01:15:56.110 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi005.front.sepia.ceph.com -o noatime 2022-04-10T01:15:56.110 DEBUG:teuthology.orchestra.run.smithi005:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2022-04-10T01:15:56.191 DEBUG:teuthology.orchestra.run.smithi005:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2022-04-10T01:15:56.261 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2022-04-10T01:15:56.329 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2022-04-10T01:15:56.329 INFO:tasks.ceph:role: cluster1.osd.2 2022-04-10T01:15:56.329 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi005.front.sepia.ceph.com 2022-04-10T01:15:56.330 DEBUG:teuthology.orchestra.run.smithi005:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2022-04-10T01:15:56.395 INFO:teuthology.orchestra.run.smithi005.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2022-04-10T01:15:56.396 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 attr=2, projid32bit=1 2022-04-10T01:15:56.396 INFO:teuthology.orchestra.run.smithi005.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-04-10T01:15:56.396 INFO:teuthology.orchestra.run.smithi005.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-04-10T01:15:56.396 INFO:teuthology.orchestra.run.smithi005.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-04-10T01:15:56.397 INFO:teuthology.orchestra.run.smithi005.stdout: = sunit=0 swidth=0 blks 2022-04-10T01:15:56.397 INFO:teuthology.orchestra.run.smithi005.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-04-10T01:15:56.397 INFO:teuthology.orchestra.run.smithi005.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-04-10T01:15:56.397 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-04-10T01:15:56.397 INFO:teuthology.orchestra.run.smithi005.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-04-10T01:15:56.418 INFO:teuthology.orchestra.run.smithi005.stdout:Discarding blocks...Done. 2022-04-10T01:15:56.420 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi005.front.sepia.ceph.com -o noatime 2022-04-10T01:15:56.420 DEBUG:teuthology.orchestra.run.smithi005:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2022-04-10T01:15:56.500 DEBUG:teuthology.orchestra.run.smithi005:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2022-04-10T01:15:56.571 DEBUG:teuthology.orchestra.run.smithi005:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2022-04-10T01:15:56.656 INFO:teuthology.orchestra.run.smithi005.stderr:2022-04-10T01:15:56.654+0000 7ff96aeef3c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2022-04-10T01:15:56.656 INFO:teuthology.orchestra.run.smithi005.stderr:2022-04-10T01:15:56.654+0000 7ff96aeef3c0 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2022-04-10T01:15:56.656 INFO:teuthology.orchestra.run.smithi005.stderr:2022-04-10T01:15:56.654+0000 7ff96aeef3c0 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2022-04-10T01:15:56.656 INFO:teuthology.orchestra.run.smithi005.stderr:2022-04-10T01:15:56.654+0000 7ff96aeef3c0 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2022-04-10T01:15:56.657 INFO:teuthology.orchestra.run.smithi005.stderr:2022-04-10T01:15:56.654+0000 7ff96aeef3c0 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2022-04-10T01:15:56.657 INFO:teuthology.orchestra.run.smithi005.stderr:2022-04-10T01:15:56.655+0000 7ff96aeef3c0 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid 2022-04-10T01:15:59.517 DEBUG:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2022-04-10T01:15:59.591 DEBUG:teuthology.orchestra.run.smithi005:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2022-04-10T01:15:59.682 INFO:teuthology.orchestra.run.smithi005.stderr:2022-04-10T01:15:59.680+0000 7f6f1778c3c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2022-04-10T01:15:59.682 INFO:teuthology.orchestra.run.smithi005.stderr:2022-04-10T01:15:59.681+0000 7f6f1778c3c0 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2022-04-10T01:15:59.683 INFO:teuthology.orchestra.run.smithi005.stderr:2022-04-10T01:15:59.681+0000 7f6f1778c3c0 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2022-04-10T01:15:59.683 INFO:teuthology.orchestra.run.smithi005.stderr:2022-04-10T01:15:59.681+0000 7f6f1778c3c0 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2022-04-10T01:15:59.683 INFO:teuthology.orchestra.run.smithi005.stderr:2022-04-10T01:15:59.681+0000 7f6f1778c3c0 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2022-04-10T01:15:59.683 INFO:teuthology.orchestra.run.smithi005.stderr:2022-04-10T01:15:59.681+0000 7f6f1778c3c0 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid 2022-04-10T01:16:02.542 DEBUG:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2022-04-10T01:16:02.575 DEBUG:teuthology.orchestra.run.smithi005:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2022-04-10T01:16:02.665 INFO:teuthology.orchestra.run.smithi005.stderr:2022-04-10T01:16:02.663+0000 7fc974c5c3c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2022-04-10T01:16:02.666 INFO:teuthology.orchestra.run.smithi005.stderr:2022-04-10T01:16:02.664+0000 7fc974c5c3c0 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2022-04-10T01:16:02.666 INFO:teuthology.orchestra.run.smithi005.stderr:2022-04-10T01:16:02.664+0000 7fc974c5c3c0 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2022-04-10T01:16:02.666 INFO:teuthology.orchestra.run.smithi005.stderr:2022-04-10T01:16:02.664+0000 7fc974c5c3c0 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2022-04-10T01:16:02.666 INFO:teuthology.orchestra.run.smithi005.stderr:2022-04-10T01:16:02.664+0000 7fc974c5c3c0 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2022-04-10T01:16:02.667 INFO:teuthology.orchestra.run.smithi005.stderr:2022-04-10T01:16:02.664+0000 7fc974c5c3c0 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid 2022-04-10T01:16:05.525 DEBUG:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2022-04-10T01:16:05.559 INFO:tasks.ceph:Reading keys from all nodes... 2022-04-10T01:16:05.560 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-04-10T01:16:05.560 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2022-04-10T01:16:05.631 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-04-10T01:16:05.631 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2022-04-10T01:16:05.701 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-04-10T01:16:05.701 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2022-04-10T01:16:05.771 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-04-10T01:16:05.771 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2022-04-10T01:16:05.840 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-04-10T01:16:05.841 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2022-04-10T01:16:05.898 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-04-10T01:16:05.899 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2022-04-10T01:16:05.920 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-04-10T01:16:05.921 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2022-04-10T01:16:05.981 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-04-10T01:16:05.981 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2022-04-10T01:16:06.042 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-04-10T01:16:06.043 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2022-04-10T01:16:06.103 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-04-10T01:16:06.103 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2022-04-10T01:16:06.163 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-04-10T01:16:06.163 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2022-04-10T01:16:06.223 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-04-10T01:16:06.224 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2022-04-10T01:16:06.283 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-04-10T01:16:06.284 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2022-04-10T01:16:06.343 INFO:tasks.ceph:Adding keys to all mons... 2022-04-10T01:16:06.343 DEBUG:teuthology.orchestra.run.smithi005:> sudo tee -a /etc/ceph/cluster1.keyring 2022-04-10T01:16:06.374 INFO:teuthology.orchestra.run.smithi005.stdout:[mgr.x] 2022-04-10T01:16:06.375 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQDKL1JiGWSVNBAAJpWmnLvgpHA44WnKFL/MBw== 2022-04-10T01:16:06.375 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.0] 2022-04-10T01:16:06.375 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQDML1JigEQSJxAAcwFLGMWEgklU55MPrKuaBA== 2022-04-10T01:16:06.376 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.1] 2022-04-10T01:16:06.376 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQDPL1JiZnKiKBAAJ3pCH8DZQ0m4ris2Uca8wQ== 2022-04-10T01:16:06.376 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.2] 2022-04-10T01:16:06.376 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQDSL1JiShueJxAA0/bVAUixRYT7oNe6npxZlQ== 2022-04-10T01:16:06.377 INFO:teuthology.orchestra.run.smithi005.stdout:[client.0] 2022-04-10T01:16:06.377 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQDKL1JiDcU5NxAAaWK5tIZ1S0fzRT/7V1XAgw== 2022-04-10T01:16:06.377 INFO:teuthology.orchestra.run.smithi005.stdout:[client.mirror] 2022-04-10T01:16:06.377 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQDLL1JiznhNARAAYHuf2qRpbH7MrKgrR3ogYA== 2022-04-10T01:16:06.378 INFO:teuthology.orchestra.run.smithi005.stdout:[client.mirror.0] 2022-04-10T01:16:06.378 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQDLL1Ji/XHDBBAAJ73+cNssDz8VWxIbkjLxvg== 2022-04-10T01:16:06.378 INFO:teuthology.orchestra.run.smithi005.stdout:[client.mirror.1] 2022-04-10T01:16:06.378 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQDLL1Ji9lmBChAAnCxgqhFGZzAGRxRm12q5sQ== 2022-04-10T01:16:06.378 INFO:teuthology.orchestra.run.smithi005.stdout:[client.mirror.2] 2022-04-10T01:16:06.379 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQDLL1Ji6HUdEBAAit9F1dyzb1snboRnN8/u/Q== 2022-04-10T01:16:06.379 INFO:teuthology.orchestra.run.smithi005.stdout:[client.mirror.3] 2022-04-10T01:16:06.379 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQDLL1JiVJ21FRAAFwn/Tx2kSz8AHUT5lq/IpQ== 2022-04-10T01:16:06.379 INFO:teuthology.orchestra.run.smithi005.stdout:[client.mirror.4] 2022-04-10T01:16:06.380 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQDLL1Ji/XdNGxAARt4ddkQDqwdUYtO+3lFStg== 2022-04-10T01:16:06.380 INFO:teuthology.orchestra.run.smithi005.stdout:[client.mirror.5] 2022-04-10T01:16:06.380 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQDLL1Ji2NruIBAAhGr9Dv8/3hp8nouTkscR/A== 2022-04-10T01:16:06.380 INFO:teuthology.orchestra.run.smithi005.stdout:[client.mirror.6] 2022-04-10T01:16:06.381 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQDLL1Ji1JGBJhAAJjCmG75byAQW/yCgIX1yNA== 2022-04-10T01:16:06.382 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2022-04-10T01:16:06.480 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-04-10T01:16:06.535 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-04-10T01:16:06.587 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-04-10T01:16:06.638 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-04-10T01:16:06.691 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-04-10T01:16:06.744 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-04-10T01:16:06.797 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-04-10T01:16:06.851 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-04-10T01:16:06.906 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-04-10T01:16:06.959 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-04-10T01:16:07.013 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-04-10T01:16:07.067 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-04-10T01:16:07.120 INFO:tasks.ceph:Running mkfs on mon nodes... 2022-04-10T01:16:07.120 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2022-04-10T01:16:07.148 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2022-04-10T01:16:07.243 DEBUG:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2022-04-10T01:16:07.272 DEBUG:teuthology.orchestra.run.smithi005:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2022-04-10T01:16:07.329 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2022-04-10T01:16:07.330 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2022-04-10T01:16:07.331 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2022-04-10T01:16:07.371 INFO:tasks.ceph.cluster1.mon.a:Started 2022-04-10T01:16:07.372 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2022-04-10T01:16:07.372 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2022-04-10T01:16:07.373 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2022-04-10T01:16:07.375 INFO:tasks.ceph.cluster1.mgr.x:Started 2022-04-10T01:16:07.375 INFO:tasks.ceph:Setting crush tunables to default 2022-04-10T01:16:07.376 DEBUG:teuthology.orchestra.run.smithi005:> sudo ceph --cluster cluster1 osd crush tunables default 2022-04-10T01:16:07.584 INFO:teuthology.orchestra.run.smithi005.stderr:adjusted tunables profile to default 2022-04-10T01:16:07.596 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2022-04-10T01:16:07.596 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-04-10T01:16:07.597 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2022-04-10T01:16:07.624 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-04-10T01:16:07.624 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2022-04-10T01:16:07.691 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-04-10T01:16:07.691 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2022-04-10T01:16:07.759 DEBUG:teuthology.orchestra.run.smithi005:> sudo ceph --cluster cluster1 osd new d8722183-c8d6-411a-b7ed-e16cc7e73a0e 0 2022-04-10T01:16:07.844 INFO:tasks.ceph.cluster1.mgr.x.smithi005.stderr:2022-04-10T01:16:07.843+0000 7fa27ffa4000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-04-10T01:16:07.919 INFO:tasks.ceph.cluster1.mgr.x.smithi005.stderr:2022-04-10T01:16:07.918+0000 7fa27ffa4000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-04-10T01:16:08.002 INFO:tasks.ceph.cluster1.mgr.x.smithi005.stderr:2022-04-10T01:16:08.001+0000 7fa27ffa4000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-04-10T01:16:08.012 INFO:teuthology.orchestra.run.smithi005.stdout:0 2022-04-10T01:16:08.025 DEBUG:teuthology.orchestra.run.smithi005:> sudo ceph --cluster cluster1 osd new 7c578a43-b257-445f-b4c9-4405d4a0c48f 1 2022-04-10T01:16:08.076 INFO:tasks.ceph.cluster1.mgr.x.smithi005.stderr:2022-04-10T01:16:08.075+0000 7fa27ffa4000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-04-10T01:16:08.144 INFO:tasks.ceph.cluster1.mgr.x.smithi005.stderr:2022-04-10T01:16:08.143+0000 7fa27ffa4000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-04-10T01:16:08.238 INFO:teuthology.orchestra.run.smithi005.stdout:1 2022-04-10T01:16:08.250 DEBUG:teuthology.orchestra.run.smithi005:> sudo ceph --cluster cluster1 osd new 50222efa-8260-46a2-83ae-4d37e9046dd7 2 2022-04-10T01:16:08.339 INFO:tasks.ceph.cluster1.mgr.x.smithi005.stderr:2022-04-10T01:16:08.338+0000 7fa27ffa4000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-04-10T01:16:08.412 INFO:tasks.ceph.cluster1.mgr.x.smithi005.stderr:2022-04-10T01:16:08.411+0000 7fa27ffa4000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-04-10T01:16:08.464 INFO:teuthology.orchestra.run.smithi005.stdout:2 2022-04-10T01:16:08.477 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2022-04-10T01:16:08.477 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2022-04-10T01:16:08.479 INFO:tasks.ceph.cluster1.osd.0:Started 2022-04-10T01:16:08.480 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2022-04-10T01:16:08.480 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2022-04-10T01:16:08.482 INFO:tasks.ceph.cluster1.osd.1:Started 2022-04-10T01:16:08.487 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2022-04-10T01:16:08.487 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2022-04-10T01:16:08.489 INFO:tasks.ceph.cluster1.osd.2:Started 2022-04-10T01:16:08.490 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2022-04-10T01:16:08.515 INFO:tasks.ceph.cluster1.mgr.x.smithi005.stderr:2022-04-10T01:16:08.515+0000 7fa27ffa4000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-04-10T01:16:08.705 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:16:08.705 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":5,"fsid":"f7ffc1b1-9015-4973-aea5-caa6dd85ed1d","created":"2022-04-10T01:16:07.486579+0000","modified":"2022-04-10T01:16:08.462818+0000","last_up_change":"0.000000","last_in_change":"2022-04-10T01:16:08.462818+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"d8722183-c8d6-411a-b7ed-e16cc7e73a0e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7c578a43-b257-445f-b4c9-4405d4a0c48f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"50222efa-8260-46a2-83ae-4d37e9046dd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-04-10T01:16:08.717 INFO:tasks.ceph.ceph_manager.cluster1:[] 2022-04-10T01:16:08.718 INFO:tasks.ceph:Waiting for OSDs to come up 2022-04-10T01:16:08.794 INFO:tasks.ceph.cluster1.mgr.x.smithi005.stderr:2022-04-10T01:16:08.792+0000 7fa27ffa4000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-04-10T01:16:08.894 INFO:tasks.ceph.cluster1.mgr.x.smithi005.stderr:2022-04-10T01:16:08.893+0000 7fa27ffa4000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-04-10T01:16:09.021 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2022-04-10T01:16:09.033 INFO:tasks.ceph.cluster1.mgr.x.smithi005.stderr:2022-04-10T01:16:09.032+0000 7fa27ffa4000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-04-10T01:16:09.094 INFO:tasks.ceph.cluster1.osd.2.smithi005.stderr:2022-04-10T01:16:09.093+0000 7f0c7b40a3c0 -1 Falling back to public interface 2022-04-10T01:16:09.103 INFO:tasks.ceph.cluster1.osd.1.smithi005.stderr:2022-04-10T01:16:09.102+0000 7f85976e83c0 -1 Falling back to public interface 2022-04-10T01:16:09.111 INFO:tasks.ceph.cluster1.osd.0.smithi005.stderr:2022-04-10T01:16:09.110+0000 7fb1a69983c0 -1 Falling back to public interface 2022-04-10T01:16:09.216 INFO:teuthology.misc.health.smithi005.stdout: 2022-04-10T01:16:09.217 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":5,"fsid":"f7ffc1b1-9015-4973-aea5-caa6dd85ed1d","created":"2022-04-10T01:16:07.486579+0000","modified":"2022-04-10T01:16:08.462818+0000","last_up_change":"0.000000","last_in_change":"2022-04-10T01:16:08.462818+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"d8722183-c8d6-411a-b7ed-e16cc7e73a0e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7c578a43-b257-445f-b4c9-4405d4a0c48f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"50222efa-8260-46a2-83ae-4d37e9046dd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-04-10T01:16:09.227 DEBUG:teuthology.misc:0 of 3 OSDs are up 2022-04-10T01:16:09.640 INFO:tasks.ceph.cluster1.mgr.x.smithi005.stderr:2022-04-10T01:16:09.639+0000 7fa27ffa4000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-04-10T01:16:09.720 INFO:tasks.ceph.cluster1.mgr.x.smithi005.stderr:2022-04-10T01:16:09.719+0000 7fa27ffa4000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-04-10T01:16:09.855 INFO:tasks.ceph.cluster1.mgr.x.smithi005.stderr:2022-04-10T01:16:09.854+0000 7fa27ffa4000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-04-10T01:16:10.086 INFO:tasks.ceph.cluster1.mgr.x.smithi005.stderr:2022-04-10T01:16:10.084+0000 7fa27ffa4000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-04-10T01:16:10.926 INFO:tasks.ceph.cluster1.mgr.x.smithi005.stderr:2022-04-10T01:16:10.925+0000 7fa27ffa4000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-04-10T01:16:11.077 INFO:tasks.ceph.cluster1.mgr.x.smithi005.stderr:2022-04-10T01:16:11.076+0000 7fa27ffa4000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-04-10T01:16:11.455 INFO:tasks.ceph.cluster1.mgr.x.smithi005.stderr:2022-04-10T01:16:11.453+0000 7fa27ffa4000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-04-10T01:16:11.739 INFO:tasks.ceph.cluster1.mgr.x.smithi005.stderr:2022-04-10T01:16:11.738+0000 7fa27ffa4000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-04-10T01:16:11.826 INFO:tasks.ceph.cluster1.mgr.x.smithi005.stderr:2022-04-10T01:16:11.825+0000 7fa27ffa4000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-04-10T01:16:11.915 INFO:tasks.ceph.cluster1.mgr.x.smithi005.stderr:2022-04-10T01:16:11.914+0000 7fa27ffa4000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-04-10T01:16:12.002 INFO:tasks.ceph.cluster1.mgr.x.smithi005.stderr:2022-04-10T01:16:12.001+0000 7fa27ffa4000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-04-10T01:16:12.281 INFO:tasks.ceph.cluster1.mgr.x.smithi005.stderr:2022-04-10T01:16:12.280+0000 7fa27ffa4000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-04-10T01:16:12.593 INFO:tasks.ceph.cluster1.osd.1.smithi005.stderr:2022-04-10T01:16:12.592+0000 7f85976e83c0 -1 osd.1 0 log_to_monitors true 2022-04-10T01:16:12.603 INFO:tasks.ceph.cluster1.osd.0.smithi005.stderr:2022-04-10T01:16:12.602+0000 7fb1a69983c0 -1 osd.0 0 log_to_monitors true 2022-04-10T01:16:12.623 INFO:tasks.ceph.cluster1.osd.2.smithi005.stderr:2022-04-10T01:16:12.622+0000 7f0c7b40a3c0 -1 osd.2 0 log_to_monitors true 2022-04-10T01:16:13.075 INFO:tasks.ceph.cluster1.mgr.x.smithi005.stderr:2022-04-10T01:16:13.073+0000 7fa27ffa4000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-04-10T01:16:15.531 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2022-04-10T01:16:15.738 INFO:tasks.ceph.cluster1.osd.1.smithi005.stderr:2022-04-10T01:16:15.737+0000 7f858fb06700 -1 osd.1 0 waiting for initial osdmap 2022-04-10T01:16:15.741 INFO:tasks.ceph.cluster1.osd.1.smithi005.stderr:2022-04-10T01:16:15.741+0000 7f8586c95700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-04-10T01:16:15.757 INFO:tasks.ceph.cluster1.osd.0.smithi005.stderr:2022-04-10T01:16:15.756+0000 7fb19d5b3700 -1 osd.0 0 waiting for initial osdmap 2022-04-10T01:16:15.761 INFO:tasks.ceph.cluster1.osd.0.smithi005.stderr:2022-04-10T01:16:15.760+0000 7fb195f45700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-04-10T01:16:15.770 INFO:tasks.ceph.cluster1.osd.2.smithi005.stderr:2022-04-10T01:16:15.769+0000 7f0c72025700 -1 osd.2 0 waiting for initial osdmap 2022-04-10T01:16:15.773 INFO:tasks.ceph.cluster1.osd.2.smithi005.stderr:2022-04-10T01:16:15.772+0000 7f0c6a9b7700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-04-10T01:16:15.915 INFO:teuthology.misc.health.smithi005.stdout: 2022-04-10T01:16:15.915 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":7,"fsid":"f7ffc1b1-9015-4973-aea5-caa6dd85ed1d","created":"2022-04-10T01:16:07.486579+0000","modified":"2022-04-10T01:16:14.459847+0000","last_up_change":"0.000000","last_in_change":"2022-04-10T01:16:08.462818+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"d8722183-c8d6-411a-b7ed-e16cc7e73a0e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7c578a43-b257-445f-b4c9-4405d4a0c48f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"50222efa-8260-46a2-83ae-4d37e9046dd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-04-10T01:16:15.927 DEBUG:teuthology.misc:0 of 3 OSDs are up 2022-04-10T01:16:22.231 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2022-04-10T01:16:22.546 INFO:teuthology.misc.health.smithi005.stdout: 2022-04-10T01:16:22.547 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":11,"fsid":"f7ffc1b1-9015-4973-aea5-caa6dd85ed1d","created":"2022-04-10T01:16:07.486579+0000","modified":"2022-04-10T01:16:19.736743+0000","last_up_change":"2022-04-10T01:16:16.734623+0000","last_in_change":"2022-04-10T01:16:08.462818+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-04-10T01:16:17.148561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"d8722183-c8d6-411a-b7ed-e16cc7e73a0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6816","nonce":33847},{"type":"v1","addr":"172.21.15.5:6817","nonce":33847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":33847},{"type":"v1","addr":"172.21.15.5:6819","nonce":33847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6822","nonce":33847},{"type":"v1","addr":"172.21.15.5:6823","nonce":33847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":33847},{"type":"v1","addr":"172.21.15.5:6821","nonce":33847}]},"public_addr":"172.21.15.5:6817/33847","cluster_addr":"172.21.15.5:6819/33847","heartbeat_back_addr":"172.21.15.5:6823/33847","heartbeat_front_addr":"172.21.15.5:6821/33847","state":["exists","up"]},{"osd":1,"uuid":"7c578a43-b257-445f-b4c9-4405d4a0c48f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":33848},{"type":"v1","addr":"172.21.15.5:6809","nonce":33848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6810","nonce":33848},{"type":"v1","addr":"172.21.15.5:6811","nonce":33848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":33848},{"type":"v1","addr":"172.21.15.5:6815","nonce":33848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":33848},{"type":"v1","addr":"172.21.15.5:6813","nonce":33848}]},"public_addr":"172.21.15.5:6809/33848","cluster_addr":"172.21.15.5:6811/33848","heartbeat_back_addr":"172.21.15.5:6815/33848","heartbeat_front_addr":"172.21.15.5:6813/33848","state":["exists","up"]},{"osd":2,"uuid":"50222efa-8260-46a2-83ae-4d37e9046dd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":33846},{"type":"v1","addr":"172.21.15.5:6801","nonce":33846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":33846},{"type":"v1","addr":"172.21.15.5:6803","nonce":33846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":33846},{"type":"v1","addr":"172.21.15.5:6807","nonce":33846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":33846},{"type":"v1","addr":"172.21.15.5:6805","nonce":33846}]},"public_addr":"172.21.15.5:6801/33846","cluster_addr":"172.21.15.5:6803/33846","heartbeat_back_addr":"172.21.15.5:6807/33846","heartbeat_front_addr":"172.21.15.5:6805/33846","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-10T01:16:13.589854+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-10T01:16:13.559497+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-10T01:16:13.630103+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-04-10T01:16:22.559 DEBUG:teuthology.misc:3 of 3 OSDs are up 2022-04-10T01:16:22.559 INFO:tasks.ceph:Creating RBD pool 2022-04-10T01:16:22.560 DEBUG:teuthology.orchestra.run.smithi005:> sudo ceph --cluster cluster1 osd pool create rbd 8 2022-04-10T01:16:23.753 INFO:teuthology.orchestra.run.smithi005.stderr:pool 'rbd' created 2022-04-10T01:16:23.767 DEBUG:teuthology.orchestra.run.smithi005:> sudo ceph --cluster cluster1 osd pool application enable rbd rbd --yes-i-really-mean-it 2022-04-10T01:16:24.758 INFO:teuthology.orchestra.run.smithi005.stderr:enabled application 'rbd' on pool 'rbd' 2022-04-10T01:16:24.771 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2022-04-10T01:16:24.771 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2022-04-10T01:16:24.772 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2022-04-10T01:16:24.772 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2022-04-10T01:16:24.773 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2022-04-10T01:16:25.142 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:16:25.155 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":5,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":33466},{"type":"v1","addr":"172.21.15.5:6825","nonce":33466}]},"active_addr":"172.21.15.5:6825/33466","active_change":"2022-04-10T01:16:13.087763+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.5:0","nonce":1858726}]},{"addrvec":[{"type":"v2","addr":"172.21.15.5:0","nonce":2784701611}]},{"addrvec":[{"type":"v2","addr":"172.21.15.5:0","nonce":75689407}]},{"addrvec":[{"type":"v2","addr":"172.21.15.5:0","nonce":1596017392}]}]}} 2022-04-10T01:16:25.158 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2022-04-10T01:16:25.158 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2022-04-10T01:16:25.159 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2022-04-10T01:16:25.469 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:16:25.470 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":13,"fsid":"f7ffc1b1-9015-4973-aea5-caa6dd85ed1d","created":"2022-04-10T01:16:07.486579+0000","modified":"2022-04-10T01:16:24.745155+0000","last_up_change":"2022-04-10T01:16:16.734623+0000","last_in_change":"2022-04-10T01:16:08.462818+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-04-10T01:16:17.148561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-04-10T01:16:22.899330+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d8722183-c8d6-411a-b7ed-e16cc7e73a0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6816","nonce":33847},{"type":"v1","addr":"172.21.15.5:6817","nonce":33847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":33847},{"type":"v1","addr":"172.21.15.5:6819","nonce":33847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6822","nonce":33847},{"type":"v1","addr":"172.21.15.5:6823","nonce":33847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":33847},{"type":"v1","addr":"172.21.15.5:6821","nonce":33847}]},"public_addr":"172.21.15.5:6817/33847","cluster_addr":"172.21.15.5:6819/33847","heartbeat_back_addr":"172.21.15.5:6823/33847","heartbeat_front_addr":"172.21.15.5:6821/33847","state":["exists","up"]},{"osd":1,"uuid":"7c578a43-b257-445f-b4c9-4405d4a0c48f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":33848},{"type":"v1","addr":"172.21.15.5:6809","nonce":33848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6810","nonce":33848},{"type":"v1","addr":"172.21.15.5:6811","nonce":33848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":33848},{"type":"v1","addr":"172.21.15.5:6815","nonce":33848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":33848},{"type":"v1","addr":"172.21.15.5:6813","nonce":33848}]},"public_addr":"172.21.15.5:6809/33848","cluster_addr":"172.21.15.5:6811/33848","heartbeat_back_addr":"172.21.15.5:6815/33848","heartbeat_front_addr":"172.21.15.5:6813/33848","state":["exists","up"]},{"osd":2,"uuid":"50222efa-8260-46a2-83ae-4d37e9046dd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":33846},{"type":"v1","addr":"172.21.15.5:6801","nonce":33846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":33846},{"type":"v1","addr":"172.21.15.5:6803","nonce":33846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":33846},{"type":"v1","addr":"172.21.15.5:6807","nonce":33846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":33846},{"type":"v1","addr":"172.21.15.5:6805","nonce":33846}]},"public_addr":"172.21.15.5:6801/33846","cluster_addr":"172.21.15.5:6803/33846","heartbeat_back_addr":"172.21.15.5:6807/33846","heartbeat_front_addr":"172.21.15.5:6805/33846","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-10T01:16:13.589854+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-10T01:16:13.559497+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-10T01:16:13.630103+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-04-10T01:16:25.483 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2022-04-10T01:16:25.483 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2022-04-10T01:16:25.793 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:16:25.793 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":14,"fsid":"f7ffc1b1-9015-4973-aea5-caa6dd85ed1d","created":"2022-04-10T01:16:07.486579+0000","modified":"2022-04-10T01:16:25.748037+0000","last_up_change":"2022-04-10T01:16:16.734623+0000","last_in_change":"2022-04-10T01:16:08.462818+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-04-10T01:16:17.148561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-04-10T01:16:22.899330+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d8722183-c8d6-411a-b7ed-e16cc7e73a0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6816","nonce":33847},{"type":"v1","addr":"172.21.15.5:6817","nonce":33847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":33847},{"type":"v1","addr":"172.21.15.5:6819","nonce":33847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6822","nonce":33847},{"type":"v1","addr":"172.21.15.5:6823","nonce":33847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":33847},{"type":"v1","addr":"172.21.15.5:6821","nonce":33847}]},"public_addr":"172.21.15.5:6817/33847","cluster_addr":"172.21.15.5:6819/33847","heartbeat_back_addr":"172.21.15.5:6823/33847","heartbeat_front_addr":"172.21.15.5:6821/33847","state":["exists","up"]},{"osd":1,"uuid":"7c578a43-b257-445f-b4c9-4405d4a0c48f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":33848},{"type":"v1","addr":"172.21.15.5:6809","nonce":33848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6810","nonce":33848},{"type":"v1","addr":"172.21.15.5:6811","nonce":33848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":33848},{"type":"v1","addr":"172.21.15.5:6815","nonce":33848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":33848},{"type":"v1","addr":"172.21.15.5:6813","nonce":33848}]},"public_addr":"172.21.15.5:6809/33848","cluster_addr":"172.21.15.5:6811/33848","heartbeat_back_addr":"172.21.15.5:6815/33848","heartbeat_front_addr":"172.21.15.5:6813/33848","state":["exists","up"]},{"osd":2,"uuid":"50222efa-8260-46a2-83ae-4d37e9046dd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":33846},{"type":"v1","addr":"172.21.15.5:6801","nonce":33846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":33846},{"type":"v1","addr":"172.21.15.5:6803","nonce":33846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":33846},{"type":"v1","addr":"172.21.15.5:6807","nonce":33846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":33846},{"type":"v1","addr":"172.21.15.5:6805","nonce":33846}]},"public_addr":"172.21.15.5:6801/33846","cluster_addr":"172.21.15.5:6803/33846","heartbeat_back_addr":"172.21.15.5:6807/33846","heartbeat_front_addr":"172.21.15.5:6805/33846","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-10T01:16:13.589854+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-10T01:16:13.559497+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-10T01:16:13.630103+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-04-10T01:16:25.807 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2022-04-10T01:16:25.808 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2022-04-10T01:16:25.808 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2022-04-10T01:16:25.974 INFO:teuthology.orchestra.run.smithi005.stdout:34359738372 2022-04-10T01:16:25.974 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2022-04-10T01:16:25.982 INFO:teuthology.orchestra.run.smithi005.stdout:34359738371 2022-04-10T01:16:25.983 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2022-04-10T01:16:25.984 INFO:teuthology.orchestra.run.smithi005.stdout:34359738372 2022-04-10T01:16:25.985 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2022-04-10T01:16:26.311 INFO:teuthology.orchestra.run.smithi005.stdout:34359738371 2022-04-10T01:16:26.325 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738371 for osd.2 2022-04-10T01:16:26.354 INFO:teuthology.orchestra.run.smithi005.stdout:34359738371 2022-04-10T01:16:26.363 INFO:teuthology.orchestra.run.smithi005.stdout:34359738370 2022-04-10T01:16:26.369 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738371 for osd.1 2022-04-10T01:16:26.377 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738370 for osd.0 2022-04-10T01:16:27.326 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2022-04-10T01:16:27.370 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2022-04-10T01:16:27.378 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2022-04-10T01:16:27.666 INFO:teuthology.orchestra.run.smithi005.stdout:34359738371 2022-04-10T01:16:27.681 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738371 for osd.2 2022-04-10T01:16:27.705 INFO:teuthology.orchestra.run.smithi005.stdout:34359738371 2022-04-10T01:16:27.719 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738371 for osd.1 2022-04-10T01:16:27.737 INFO:teuthology.orchestra.run.smithi005.stdout:34359738370 2022-04-10T01:16:27.834 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738370 for osd.0 2022-04-10T01:16:28.682 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2022-04-10T01:16:28.720 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2022-04-10T01:16:28.835 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2022-04-10T01:16:29.021 INFO:teuthology.orchestra.run.smithi005.stdout:34359738372 2022-04-10T01:16:29.041 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738372 for osd.2 2022-04-10T01:16:29.041 DEBUG:teuthology.parallel:result is None 2022-04-10T01:16:29.054 INFO:teuthology.orchestra.run.smithi005.stdout:34359738372 2022-04-10T01:16:29.068 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738372 for osd.1 2022-04-10T01:16:29.068 DEBUG:teuthology.parallel:result is None 2022-04-10T01:16:29.163 INFO:teuthology.orchestra.run.smithi005.stdout:34359738371 2022-04-10T01:16:29.177 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738371 for osd.0 2022-04-10T01:16:29.177 DEBUG:teuthology.parallel:result is None 2022-04-10T01:16:29.178 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2022-04-10T01:16:29.178 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2022-04-10T01:16:29.481 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:16:29.483 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2022-04-10T01:16:29.494 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2022-04-10T01:16:29.094141+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":37012,"kb_used_data":1276,"kb_used_omap":0,"kb_used_meta":35712,"kb_avail":283078508,"statfs":{"total":289910292480,"available":289872392192,"internally_reserved":0,"allocated":1306624,"data_stored":1037014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":36569088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001235"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-04-10T01:16:24.757999+0000","last_change":"2022-04-10T01:16:24.757033+0000","last_active":"2022-04-10T01:16:24.757999+0000","last_peered":"2022-04-10T01:16:24.757999+0000","last_clean":"2022-04-10T01:16:24.757999+0000","last_became_active":"2022-04-10T01:16:24.756694+0000","last_became_peered":"2022-04-10T01:16:24.756694+0000","last_unstale":"2022-04-10T01:16:24.757999+0000","last_undegraded":"2022-04-10T01:16:24.757999+0000","last_fullsized":"2022-04-10T01:16:24.757999+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-10T01:16:23.742429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-10T01:16:23.742429+0000","last_clean_scrub_stamp":"2022-04-10T01:16:23.742429+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-11T03:02:04.670400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-04-10T01:16:24.756577+0000","last_change":"2022-04-10T01:16:24.754586+0000","last_active":"2022-04-10T01:16:24.756577+0000","last_peered":"2022-04-10T01:16:24.756577+0000","last_clean":"2022-04-10T01:16:24.756577+0000","last_became_active":"2022-04-10T01:16:24.754281+0000","last_became_peered":"2022-04-10T01:16:24.754281+0000","last_unstale":"2022-04-10T01:16:24.756577+0000","last_undegraded":"2022-04-10T01:16:24.756577+0000","last_fullsized":"2022-04-10T01:16:24.756577+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-10T01:16:23.742429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-10T01:16:23.742429+0000","last_clean_scrub_stamp":"2022-04-10T01:16:23.742429+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-11T05:14:22.428898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-04-10T01:16:24.758025+0000","last_change":"2022-04-10T01:16:24.757117+0000","last_active":"2022-04-10T01:16:24.758025+0000","last_peered":"2022-04-10T01:16:24.758025+0000","last_clean":"2022-04-10T01:16:24.758025+0000","last_became_active":"2022-04-10T01:16:24.756769+0000","last_became_peered":"2022-04-10T01:16:24.756769+0000","last_unstale":"2022-04-10T01:16:24.758025+0000","last_undegraded":"2022-04-10T01:16:24.758025+0000","last_fullsized":"2022-04-10T01:16:24.758025+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-10T01:16:23.742429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-10T01:16:23.742429+0000","last_clean_scrub_stamp":"2022-04-10T01:16:23.742429+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-11T03:13:47.935767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-04-10T01:16:24.756149+0000","last_change":"2022-04-10T01:16:24.755067+0000","last_active":"2022-04-10T01:16:24.756149+0000","last_peered":"2022-04-10T01:16:24.756149+0000","last_clean":"2022-04-10T01:16:24.756149+0000","last_became_active":"2022-04-10T01:16:24.754829+0000","last_became_peered":"2022-04-10T01:16:24.754829+0000","last_unstale":"2022-04-10T01:16:24.756149+0000","last_undegraded":"2022-04-10T01:16:24.756149+0000","last_fullsized":"2022-04-10T01:16:24.756149+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-10T01:16:23.742429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-10T01:16:23.742429+0000","last_clean_scrub_stamp":"2022-04-10T01:16:23.742429+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-11T07:47:58.373201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":139,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-04-10T01:16:24.752817+0000","last_change":"2022-04-10T01:16:19.624845+0000","last_active":"2022-04-10T01:16:24.752817+0000","last_peered":"2022-04-10T01:16:24.752817+0000","last_clean":"2022-04-10T01:16:24.752817+0000","last_became_active":"2022-04-10T01:16:19.623665+0000","last_became_peered":"2022-04-10T01:16:19.623665+0000","last_unstale":"2022-04-10T01:16:24.752817+0000","last_undegraded":"2022-04-10T01:16:24.752817+0000","last_fullsized":"2022-04-10T01:16:24.752817+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-10T01:16:17.735372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-10T01:16:17.735372+0000","last_clean_scrub_stamp":"2022-04-10T01:16:17.735372+0000","objects_scrubbed":0,"log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-11T10:48:17.777157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-04-10T01:16:24.756514+0000","last_change":"2022-04-10T01:16:24.754068+0000","last_active":"2022-04-10T01:16:24.756514+0000","last_peered":"2022-04-10T01:16:24.756514+0000","last_clean":"2022-04-10T01:16:24.756514+0000","last_became_active":"2022-04-10T01:16:24.753719+0000","last_became_peered":"2022-04-10T01:16:24.753719+0000","last_unstale":"2022-04-10T01:16:24.756514+0000","last_undegraded":"2022-04-10T01:16:24.756514+0000","last_fullsized":"2022-04-10T01:16:24.756514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-10T01:16:23.742429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-10T01:16:23.742429+0000","last_clean_scrub_stamp":"2022-04-10T01:16:23.742429+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-11T02:48:11.125896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-04-10T01:16:24.756147+0000","last_change":"2022-04-10T01:16:24.754752+0000","last_active":"2022-04-10T01:16:24.756147+0000","last_peered":"2022-04-10T01:16:24.756147+0000","last_clean":"2022-04-10T01:16:24.756147+0000","last_became_active":"2022-04-10T01:16:24.754410+0000","last_became_peered":"2022-04-10T01:16:24.754410+0000","last_unstale":"2022-04-10T01:16:24.756147+0000","last_undegraded":"2022-04-10T01:16:24.756147+0000","last_fullsized":"2022-04-10T01:16:24.756147+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-10T01:16:23.742429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-10T01:16:23.742429+0000","last_clean_scrub_stamp":"2022-04-10T01:16:23.742429+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-11T07:07:04.990249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-04-10T01:16:24.756476+0000","last_change":"2022-04-10T01:16:24.755112+0000","last_active":"2022-04-10T01:16:24.756476+0000","last_peered":"2022-04-10T01:16:24.756476+0000","last_clean":"2022-04-10T01:16:24.756476+0000","last_became_active":"2022-04-10T01:16:24.754818+0000","last_became_peered":"2022-04-10T01:16:24.754818+0000","last_unstale":"2022-04-10T01:16:24.756476+0000","last_undegraded":"2022-04-10T01:16:24.756476+0000","last_fullsized":"2022-04-10T01:16:24.756476+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-10T01:16:23.742429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-10T01:16:23.742429+0000","last_clean_scrub_stamp":"2022-04-10T01:16:23.742429+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-11T10:53:57.952507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-04-10T01:16:24.758027+0000","last_change":"2022-04-10T01:16:24.757221+0000","last_active":"2022-04-10T01:16:24.758027+0000","last_peered":"2022-04-10T01:16:24.758027+0000","last_clean":"2022-04-10T01:16:24.758027+0000","last_became_active":"2022-04-10T01:16:24.756732+0000","last_became_peered":"2022-04-10T01:16:24.756732+0000","last_unstale":"2022-04-10T01:16:24.758027+0000","last_undegraded":"2022-04-10T01:16:24.758027+0000","last_fullsized":"2022-04-10T01:16:24.758027+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-10T01:16:23.742429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-10T01:16:23.742429+0000","last_clean_scrub_stamp":"2022-04-10T01:16:23.742429+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-11T02:40:02.169345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":933912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11776,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":11648,"kb_avail":94360064,"statfs":{"total":96636764160,"available":96624705536,"internally_reserved":0,"allocated":122880,"data_stored":34621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":11927552},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27900000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27000000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12616,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":12032,"kb_avail":94359224,"statfs":{"total":96636764160,"available":96623845376,"internally_reserved":0,"allocated":589824,"data_stored":500816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":12320768},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.31900000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12620,"kb_used_data":580,"kb_used_omap":0,"kb_used_meta":12032,"kb_avail":94359220,"statfs":{"total":96636764160,"available":96623841280,"internally_reserved":0,"allocated":593920,"data_stored":501577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":12320768},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-04-10T01:16:29.495 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2022-04-10T01:16:29.801 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:16:29.803 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2022-04-10T01:16:29.814 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2022-04-10T01:16:29.094141+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":37012,"kb_used_data":1276,"kb_used_omap":0,"kb_used_meta":35712,"kb_avail":283078508,"statfs":{"total":289910292480,"available":289872392192,"internally_reserved":0,"allocated":1306624,"data_stored":1037014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":36569088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001235"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-04-10T01:16:24.757999+0000","last_change":"2022-04-10T01:16:24.757033+0000","last_active":"2022-04-10T01:16:24.757999+0000","last_peered":"2022-04-10T01:16:24.757999+0000","last_clean":"2022-04-10T01:16:24.757999+0000","last_became_active":"2022-04-10T01:16:24.756694+0000","last_became_peered":"2022-04-10T01:16:24.756694+0000","last_unstale":"2022-04-10T01:16:24.757999+0000","last_undegraded":"2022-04-10T01:16:24.757999+0000","last_fullsized":"2022-04-10T01:16:24.757999+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-10T01:16:23.742429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-10T01:16:23.742429+0000","last_clean_scrub_stamp":"2022-04-10T01:16:23.742429+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-11T03:02:04.670400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-04-10T01:16:24.756577+0000","last_change":"2022-04-10T01:16:24.754586+0000","last_active":"2022-04-10T01:16:24.756577+0000","last_peered":"2022-04-10T01:16:24.756577+0000","last_clean":"2022-04-10T01:16:24.756577+0000","last_became_active":"2022-04-10T01:16:24.754281+0000","last_became_peered":"2022-04-10T01:16:24.754281+0000","last_unstale":"2022-04-10T01:16:24.756577+0000","last_undegraded":"2022-04-10T01:16:24.756577+0000","last_fullsized":"2022-04-10T01:16:24.756577+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-10T01:16:23.742429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-10T01:16:23.742429+0000","last_clean_scrub_stamp":"2022-04-10T01:16:23.742429+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-11T05:14:22.428898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-04-10T01:16:24.758025+0000","last_change":"2022-04-10T01:16:24.757117+0000","last_active":"2022-04-10T01:16:24.758025+0000","last_peered":"2022-04-10T01:16:24.758025+0000","last_clean":"2022-04-10T01:16:24.758025+0000","last_became_active":"2022-04-10T01:16:24.756769+0000","last_became_peered":"2022-04-10T01:16:24.756769+0000","last_unstale":"2022-04-10T01:16:24.758025+0000","last_undegraded":"2022-04-10T01:16:24.758025+0000","last_fullsized":"2022-04-10T01:16:24.758025+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-10T01:16:23.742429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-10T01:16:23.742429+0000","last_clean_scrub_stamp":"2022-04-10T01:16:23.742429+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-11T03:13:47.935767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-04-10T01:16:24.756149+0000","last_change":"2022-04-10T01:16:24.755067+0000","last_active":"2022-04-10T01:16:24.756149+0000","last_peered":"2022-04-10T01:16:24.756149+0000","last_clean":"2022-04-10T01:16:24.756149+0000","last_became_active":"2022-04-10T01:16:24.754829+0000","last_became_peered":"2022-04-10T01:16:24.754829+0000","last_unstale":"2022-04-10T01:16:24.756149+0000","last_undegraded":"2022-04-10T01:16:24.756149+0000","last_fullsized":"2022-04-10T01:16:24.756149+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-10T01:16:23.742429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-10T01:16:23.742429+0000","last_clean_scrub_stamp":"2022-04-10T01:16:23.742429+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-11T07:47:58.373201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":139,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-04-10T01:16:24.752817+0000","last_change":"2022-04-10T01:16:19.624845+0000","last_active":"2022-04-10T01:16:24.752817+0000","last_peered":"2022-04-10T01:16:24.752817+0000","last_clean":"2022-04-10T01:16:24.752817+0000","last_became_active":"2022-04-10T01:16:19.623665+0000","last_became_peered":"2022-04-10T01:16:19.623665+0000","last_unstale":"2022-04-10T01:16:24.752817+0000","last_undegraded":"2022-04-10T01:16:24.752817+0000","last_fullsized":"2022-04-10T01:16:24.752817+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-10T01:16:17.735372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-10T01:16:17.735372+0000","last_clean_scrub_stamp":"2022-04-10T01:16:17.735372+0000","objects_scrubbed":0,"log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-11T10:48:17.777157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-04-10T01:16:24.756514+0000","last_change":"2022-04-10T01:16:24.754068+0000","last_active":"2022-04-10T01:16:24.756514+0000","last_peered":"2022-04-10T01:16:24.756514+0000","last_clean":"2022-04-10T01:16:24.756514+0000","last_became_active":"2022-04-10T01:16:24.753719+0000","last_became_peered":"2022-04-10T01:16:24.753719+0000","last_unstale":"2022-04-10T01:16:24.756514+0000","last_undegraded":"2022-04-10T01:16:24.756514+0000","last_fullsized":"2022-04-10T01:16:24.756514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-10T01:16:23.742429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-10T01:16:23.742429+0000","last_clean_scrub_stamp":"2022-04-10T01:16:23.742429+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-11T02:48:11.125896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-04-10T01:16:24.756147+0000","last_change":"2022-04-10T01:16:24.754752+0000","last_active":"2022-04-10T01:16:24.756147+0000","last_peered":"2022-04-10T01:16:24.756147+0000","last_clean":"2022-04-10T01:16:24.756147+0000","last_became_active":"2022-04-10T01:16:24.754410+0000","last_became_peered":"2022-04-10T01:16:24.754410+0000","last_unstale":"2022-04-10T01:16:24.756147+0000","last_undegraded":"2022-04-10T01:16:24.756147+0000","last_fullsized":"2022-04-10T01:16:24.756147+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-10T01:16:23.742429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-10T01:16:23.742429+0000","last_clean_scrub_stamp":"2022-04-10T01:16:23.742429+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-11T07:07:04.990249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-04-10T01:16:24.756476+0000","last_change":"2022-04-10T01:16:24.755112+0000","last_active":"2022-04-10T01:16:24.756476+0000","last_peered":"2022-04-10T01:16:24.756476+0000","last_clean":"2022-04-10T01:16:24.756476+0000","last_became_active":"2022-04-10T01:16:24.754818+0000","last_became_peered":"2022-04-10T01:16:24.754818+0000","last_unstale":"2022-04-10T01:16:24.756476+0000","last_undegraded":"2022-04-10T01:16:24.756476+0000","last_fullsized":"2022-04-10T01:16:24.756476+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-10T01:16:23.742429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-10T01:16:23.742429+0000","last_clean_scrub_stamp":"2022-04-10T01:16:23.742429+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-11T10:53:57.952507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2022-04-10T01:16:24.758027+0000","last_change":"2022-04-10T01:16:24.757221+0000","last_active":"2022-04-10T01:16:24.758027+0000","last_peered":"2022-04-10T01:16:24.758027+0000","last_clean":"2022-04-10T01:16:24.758027+0000","last_became_active":"2022-04-10T01:16:24.756732+0000","last_became_peered":"2022-04-10T01:16:24.756732+0000","last_unstale":"2022-04-10T01:16:24.758027+0000","last_undegraded":"2022-04-10T01:16:24.758027+0000","last_fullsized":"2022-04-10T01:16:24.758027+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-10T01:16:23.742429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-10T01:16:23.742429+0000","last_clean_scrub_stamp":"2022-04-10T01:16:23.742429+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-11T02:40:02.169345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":933912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11776,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":11648,"kb_avail":94360064,"statfs":{"total":96636764160,"available":96624705536,"internally_reserved":0,"allocated":122880,"data_stored":34621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":11927552},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27900000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27000000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12616,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":12032,"kb_avail":94359224,"statfs":{"total":96636764160,"available":96623845376,"internally_reserved":0,"allocated":589824,"data_stored":500816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":12320768},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.31900000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12620,"kb_used_data":580,"kb_used_omap":0,"kb_used_meta":12032,"kb_avail":94359220,"statfs":{"total":96636764160,"available":96623841280,"internally_reserved":0,"allocated":593920,"data_stored":501577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":12320768},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-04-10T01:16:29.815 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2022-04-10T01:16:29.815 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2022-04-10T01:16:29.816 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2022-04-10T01:16:29.816 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2022-04-10T01:16:30.164 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:16:30.165 INFO:teuthology.orchestra.run.smithi005.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-04-10T01:16:30.178 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2022-04-10T01:16:30.178 INFO:teuthology.run_tasks:Running task ceph... 2022-04-10T01:16:30.193 INFO:tasks.ceph:Creating ceph cluster cluster2... 2022-04-10T01:16:30.194 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2022-04-10T01:16:30.194 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/mrfreezeex-2022-04-09_15:53:21-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784449', 'branch': 'wip-mrfreezeex-multipeer', 'description': 'rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} msgr-failures/few objectstore/bluestore-hybrid policy/simple supported-random-distro$/{centos_8}}', 'email': 'arthur.outhenin-chalandre@cern.ch', 'first_in_suite': False, 'job_id': '6784449', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'mrfreezeex-2022-04-09_15:53:21-rbd-wip-mrfreezeex-multipeer-distro-default-smithi', 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-mrfreezeex-multipeer'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '9b3739e792ef182b79445b926c7b7da461c05f52'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '9b3739e792ef182b79445b926c7b7da461c05f52'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-mrfreezeex-multipeer', 'sha1': '9b3739e792ef182b79445b926c7b7da461c05f52'}}, 'owner': 'scheduled_mrfreezeex@teuthology', 'priority': 165, 'repo': 'git://git.ceph.com/ceph-ci.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'sha1': '9b3739e792ef182b79445b926c7b7da461c05f52', 'sleep_before_teardown': 0, 'suite': 'rbd', 'suite_branch': 'wip-mrfreezeex-multipeer', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_9b3739e792ef182b79445b926c7b7da461c05f52/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': '9b3739e792ef182b79445b926c7b7da461c05f52', 'targets': {'smithi005.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6a7IpOZPtZwIOOYNtsNKq1njcAjwzrK01ZhS9QRS+ifxFUZFe8h8edywJ1zqQAOZRNH+E+aKD6C1F+nyv2HdWrjIXqbyTlgyYxU/VS9RHc27G2rFDS9p7aogFlkCDXHKfx+BiYqnbGBmqtGKL0Os0Ur7m1uZ0hn34qmSiZS0XD4/wSTSwW6XwhkJCMxuMaoWzthqmojoClGBbutqSDhmOsLBOf3g+7PchS3HOPGYIUJ+rd9p2lkM+3YV76Lo/9Sj5TrYxONbGx8tZEnw+xkYwrGqTbxTAax8DX7Bws8jRXQeKekmgA1m+JzxmWB7vTIx2Nm9wo0iMLWQT53nbhx2Mcjg224BSYyYvQiJnPX46yv25nTVAzLnVXQ8lxTUXmacy2qwHRyOfe8BNYCisGmU5Z9suX/pFgnFEBbzf5wcFmW/KwsuTDa/A2nzRNRpsaWBv53ZA7jNXBXIYOYoaV3GFYLA2AdhvsGTMzpXYxa9f+QUxhJWFvLiDmBJk0kcwhCk=', 'smithi043.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxbBDJqyiaOu9Bbtq4sC6enMBXqfpHtMB96N/mf2lA/aaKGlTMNpp2Zye0gQ9z4HygetquzjtuApbZg+ldNp1ZyvL7lcYvBos5rNC2+zTYmhk6+9bvHmSv/51wk67aheDavAy1QvcABfu4F/qTSGfqoWAXv+BZXYH2KjS8M7Rrp9YbBjcoGRsGwwvAzHLqcK806WaVGWOJCdfcpsKWVrL3R/MbUZRVhvsi8PvAeGzMAnpsYO5nyNH2Syla720zcmld++unoK06P4fWkF8hFOq/ooREjY6oH+R5zVohi1pDrEAviod6TLCvAhauFyhBgT7bcCvLk2P3/VvC7CkiX17A4fLZYnMZpQHfgsSLvlbrLFgF767L2r+/ekL114Zhxc4EolKhuZlEodiLAvt3LVgb3FcGq+J68W/ejuEY9nmZEl4pfa/jsq5LVTFoBwjcpJd40J8QOcPMteeniG7rH77A+ScsxTVCvAhiSiLpwxNfZjU6FkwakxUJq+3gEjzy21s='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '9b3739e792ef182b79445b926c7b7da461c05f52'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '9b3739e792ef182b79445b926c7b7da461c05f52'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '9b3739e792ef182b79445b926c7b7da461c05f52'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_IMAGE_MODE': 'snapshot', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology_branch': 'master', 'teuthology_sha1': 'a347b2ac5400a43ba76a678399b1deda7dae1a6c', 'timestamp': '2022-04-09_15:53:21', 'tube': 'smithi', 'user': 'mrfreezeex', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.22301'} 2022-04-10T01:16:30.195 DEBUG:teuthology.orchestra.run.smithi005:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2022-04-10T01:16:30.198 DEBUG:teuthology.orchestra.run.smithi043:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2022-04-10T01:16:30.219 DEBUG:teuthology.orchestra.run.smithi005:> sudo install -d -m0777 -- /var/run/ceph 2022-04-10T01:16:30.255 DEBUG:teuthology.orchestra.run.smithi043:> sudo install -d -m0777 -- /var/run/ceph 2022-04-10T01:16:30.292 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-04-10T01:16:30.292 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/scratch_devs of=/dev/stdout 2022-04-10T01:16:30.352 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-04-10T01:16:30.352 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-04-10T01:16:30.411 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-04-10T01:16:30.412 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-04-10T01:16:30.412 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 6h/6d Inode: 147431 Links: 1 2022-04-10T01:16:30.412 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-04-10T01:16:30.413 INFO:teuthology.orchestra.run.smithi043.stdout:Context: system_u:object_r:device_t:s0 2022-04-10T01:16:30.413 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2022-04-10 01:15:41.500210050 +0000 2022-04-10T01:16:30.413 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2022-04-10 01:12:30.755273223 +0000 2022-04-10T01:16:30.413 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2022-04-10 01:12:30.755273223 +0000 2022-04-10T01:16:30.413 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: - 2022-04-10T01:16:30.423 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2022-04-10T01:16:30.423 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2022-04-10T01:16:30.423 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 0.00011871 s, 4.3 MB/s 2022-04-10T01:16:30.427 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-04-10T01:16:30.486 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-04-10T01:16:30.487 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-04-10T01:16:30.487 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 6h/6d Inode: 150646 Links: 1 2022-04-10T01:16:30.487 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-04-10T01:16:30.487 INFO:teuthology.orchestra.run.smithi043.stdout:Context: system_u:object_r:device_t:s0 2022-04-10T01:16:30.488 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2022-04-10 01:15:41.500210050 +0000 2022-04-10T01:16:30.488 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2022-04-10 01:12:31.111269052 +0000 2022-04-10T01:16:30.488 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2022-04-10 01:12:31.111269052 +0000 2022-04-10T01:16:30.488 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: - 2022-04-10T01:16:30.497 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2022-04-10T01:16:30.498 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2022-04-10T01:16:30.498 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 0.000121147 s, 4.2 MB/s 2022-04-10T01:16:30.501 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-04-10T01:16:30.560 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-04-10T01:16:30.560 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-04-10T01:16:30.560 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 6h/6d Inode: 152645 Links: 1 2022-04-10T01:16:30.560 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-04-10T01:16:30.561 INFO:teuthology.orchestra.run.smithi043.stdout:Context: system_u:object_r:device_t:s0 2022-04-10T01:16:30.561 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2022-04-10 01:15:41.500210050 +0000 2022-04-10T01:16:30.561 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2022-04-10 01:12:31.471264834 +0000 2022-04-10T01:16:30.561 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2022-04-10 01:12:31.471264834 +0000 2022-04-10T01:16:30.562 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: - 2022-04-10T01:16:30.571 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2022-04-10T01:16:30.571 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2022-04-10T01:16:30.571 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 0.000107752 s, 4.8 MB/s 2022-04-10T01:16:30.574 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-04-10T01:16:30.632 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-04-10T01:16:30.633 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-04-10T01:16:30.633 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 6h/6d Inode: 151901 Links: 1 2022-04-10T01:16:30.633 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-04-10T01:16:30.633 INFO:teuthology.orchestra.run.smithi043.stdout:Context: system_u:object_r:device_t:s0 2022-04-10T01:16:30.634 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2022-04-10 01:15:41.500210050 +0000 2022-04-10T01:16:30.634 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2022-04-10 01:12:31.792261073 +0000 2022-04-10T01:16:30.634 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2022-04-10 01:12:31.792261073 +0000 2022-04-10T01:16:30.634 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: - 2022-04-10T01:16:30.643 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2022-04-10T01:16:30.643 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2022-04-10T01:16:30.643 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 9.6736e-05 s, 5.3 MB/s 2022-04-10T01:16:30.646 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2022-04-10T01:16:30.646 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi043.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2022-04-10T01:16:30.647 INFO:tasks.ceph:Generating config... 2022-04-10T01:16:30.648 INFO:tasks.ceph:[client] debug rbd = 20 2022-04-10T01:16:30.649 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2022-04-10T01:16:30.649 INFO:tasks.ceph:[client] log to stderr = False 2022-04-10T01:16:30.649 INFO:tasks.ceph:[client] rbd default features = 125 2022-04-10T01:16:30.649 INFO:tasks.ceph:[client] rbd mirror image policy type = simple 2022-04-10T01:16:30.650 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2022-04-10T01:16:30.650 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2022-04-10T01:16:30.650 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2022-04-10T01:16:30.651 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2022-04-10T01:16:30.651 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2022-04-10T01:16:30.651 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2022-04-10T01:16:30.652 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2022-04-10T01:16:30.652 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2022-04-10T01:16:30.652 INFO:tasks.ceph:[global] mon client directed command retry = 5 2022-04-10T01:16:30.653 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2022-04-10T01:16:30.653 INFO:tasks.ceph:[mgr] debug mgr = 20 2022-04-10T01:16:30.653 INFO:tasks.ceph:[mgr] debug ms = 1 2022-04-10T01:16:30.653 INFO:tasks.ceph:[mon] debug mon = 20 2022-04-10T01:16:30.654 INFO:tasks.ceph:[mon] debug ms = 1 2022-04-10T01:16:30.654 INFO:tasks.ceph:[mon] debug paxos = 20 2022-04-10T01:16:30.654 INFO:tasks.ceph:[osd] bluefs allocator = hybrid 2022-04-10T01:16:30.655 INFO:tasks.ceph:[osd] bluestore allocator = hybrid 2022-04-10T01:16:30.655 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2022-04-10T01:16:30.655 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2022-04-10T01:16:30.656 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2022-04-10T01:16:30.656 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2022-04-10T01:16:30.656 INFO:tasks.ceph:[osd] debug ms = 1 2022-04-10T01:16:30.656 INFO:tasks.ceph:[osd] debug osd = 20 2022-04-10T01:16:30.657 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2022-04-10T01:16:30.657 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2022-04-10T01:16:30.657 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2022-04-10T01:16:30.658 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2022-04-10T01:16:30.658 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2022-04-10T01:16:30.658 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2022-04-10T01:16:30.659 INFO:tasks.ceph:Setting up cluster2.mon.a... 2022-04-10T01:16:30.659 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2022-04-10T01:16:30.728 INFO:teuthology.orchestra.run.smithi043.stdout:creating /etc/ceph/cluster2.keyring 2022-04-10T01:16:30.731 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2022-04-10T01:16:30.815 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2022-04-10T01:16:30.882 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '172.21.15.43')] 2022-04-10T01:16:30.883 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.43', 'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2022-04-10T01:16:30.884 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-04-10T01:16:30.885 DEBUG:teuthology.orchestra.run.smithi043:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2022-04-10T01:16:30.939 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.43 --print /home/ubuntu/cephtest/cluster2.monmap 2022-04-10T01:16:31.017 INFO:teuthology.orchestra.run.smithi043.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2022-04-10T01:16:31.018 INFO:teuthology.orchestra.run.smithi043.stdout:monmaptool: generated fsid 1f0791d0-116c-427c-a4dc-de372595f89c 2022-04-10T01:16:31.018 INFO:teuthology.orchestra.run.smithi043.stdout:setting min_mon_release = octopus 2022-04-10T01:16:31.018 INFO:teuthology.orchestra.run.smithi043.stdout:epoch 0 2022-04-10T01:16:31.018 INFO:teuthology.orchestra.run.smithi043.stdout:fsid 1f0791d0-116c-427c-a4dc-de372595f89c 2022-04-10T01:16:31.019 INFO:teuthology.orchestra.run.smithi043.stdout:last_changed 2022-04-10T01:16:31.017402+0000 2022-04-10T01:16:31.019 INFO:teuthology.orchestra.run.smithi043.stdout:created 2022-04-10T01:16:31.017402+0000 2022-04-10T01:16:31.019 INFO:teuthology.orchestra.run.smithi043.stdout:min_mon_release 15 (octopus) 2022-04-10T01:16:31.019 INFO:teuthology.orchestra.run.smithi043.stdout:election_strategy: 1 2022-04-10T01:16:31.019 INFO:teuthology.orchestra.run.smithi043.stdout:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2022-04-10T01:16:31.020 INFO:teuthology.orchestra.run.smithi043.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2022-04-10T01:16:31.021 DEBUG:teuthology.orchestra.run.smithi043:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2022-04-10T01:16:31.078 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID 1f0791d0-116c-427c-a4dc-de372595f89c... 2022-04-10T01:16:31.079 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2022-04-10T01:16:31.082 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2022-04-10T01:16:31.135 INFO:teuthology.orchestra.run.smithi005.stdout:[global] 2022-04-10T01:16:31.135 INFO:teuthology.orchestra.run.smithi005.stdout: chdir = "" 2022-04-10T01:16:31.136 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-04-10T01:16:31.136 INFO:teuthology.orchestra.run.smithi005.stdout: auth supported = cephx 2022-04-10T01:16:31.136 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:16:31.136 INFO:teuthology.orchestra.run.smithi005.stdout: filestore xattr use omap = true 2022-04-10T01:16:31.137 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:16:31.137 INFO:teuthology.orchestra.run.smithi005.stdout: mon clock drift allowed = 1.000 2022-04-10T01:16:31.137 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:16:31.137 INFO:teuthology.orchestra.run.smithi005.stdout: osd crush chooseleaf type = 0 2022-04-10T01:16:31.137 INFO:teuthology.orchestra.run.smithi005.stdout: auth debug = true 2022-04-10T01:16:31.138 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:16:31.138 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on old message = true 2022-04-10T01:16:31.138 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on bug = true 2022-04-10T01:16:31.138 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:16:31.139 INFO:teuthology.orchestra.run.smithi005.stdout: mon max pg per osd = 10000 # >= luminous 2022-04-10T01:16:31.139 INFO:teuthology.orchestra.run.smithi005.stdout: mon pg warn max object skew = 0 2022-04-10T01:16:31.139 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:16:31.139 INFO:teuthology.orchestra.run.smithi005.stdout: # disable pg_autoscaler by default for new pools 2022-04-10T01:16:31.139 INFO:teuthology.orchestra.run.smithi005.stdout: osd_pool_default_pg_autoscale_mode = off 2022-04-10T01:16:31.140 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:16:31.140 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default size = 2 2022-04-10T01:16:31.140 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:16:31.140 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow primary affinity = true 2022-04-10T01:16:31.141 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow pg remap = true 2022-04-10T01:16:31.141 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on legacy crush tunables = false 2022-04-10T01:16:31.141 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on crush straw calc version zero = false 2022-04-10T01:16:31.141 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on no sortbitwise = false 2022-04-10T01:16:31.141 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on osd down out interval zero = false 2022-04-10T01:16:31.142 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on too few osds = false 2022-04-10T01:16:31.142 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-04-10T01:16:31.142 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_no_redundancy = false 2022-04-10T01:16:31.142 INFO:teuthology.orchestra.run.smithi005.stdout: mon_allow_pool_size_one = true 2022-04-10T01:16:31.143 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:16:31.143 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2022-04-10T01:16:31.143 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:16:31.143 INFO:teuthology.orchestra.run.smithi005.stdout: osd default data pool replay window = 5 2022-04-10T01:16:31.143 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:16:31.144 INFO:teuthology.orchestra.run.smithi005.stdout: mon allow pool delete = true 2022-04-10T01:16:31.144 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:16:31.144 INFO:teuthology.orchestra.run.smithi005.stdout: mon cluster log file level = debug 2022-04-10T01:16:31.144 INFO:teuthology.orchestra.run.smithi005.stdout: debug asserts on shutdown = true 2022-04-10T01:16:31.144 INFO:teuthology.orchestra.run.smithi005.stdout: mon health detail to clog = false 2022-04-10T01:16:31.145 INFO:teuthology.orchestra.run.smithi005.stdout: mon host = 172.21.15.43 2022-04-10T01:16:31.145 INFO:teuthology.orchestra.run.smithi005.stdout: mon client directed command retry = 5 2022-04-10T01:16:31.145 INFO:teuthology.orchestra.run.smithi005.stdout: ms inject socket failures = 5000 2022-04-10T01:16:31.145 INFO:teuthology.orchestra.run.smithi005.stdout: fsid = 1f0791d0-116c-427c-a4dc-de372595f89c 2022-04-10T01:16:31.145 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:16:31.146 INFO:teuthology.orchestra.run.smithi005.stdout:[osd] 2022-04-10T01:16:31.146 INFO:teuthology.orchestra.run.smithi005.stdout: osd journal size = 100 2022-04-10T01:16:31.146 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:16:31.146 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub load threshold = 5.0 2022-04-10T01:16:31.147 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub max interval = 600 2022-04-10T01:16:31.147 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:16:31.147 INFO:teuthology.orchestra.run.smithi005.stdout: osd recover clone overlap = true 2022-04-10T01:16:31.147 INFO:teuthology.orchestra.run.smithi005.stdout: osd recovery max chunk = 1048576 2022-04-10T01:16:31.148 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:16:31.148 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug shutdown = true 2022-04-10T01:16:31.148 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug op order = true 2022-04-10T01:16:31.148 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug verify stray on activate = true 2022-04-10T01:16:31.148 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:16:31.149 INFO:teuthology.orchestra.run.smithi005.stdout: osd open classes on start = true 2022-04-10T01:16:31.149 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug pg log writeout = true 2022-04-10T01:16:31.149 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:16:31.149 INFO:teuthology.orchestra.run.smithi005.stdout: osd deep scrub update digest min age = 30 2022-04-10T01:16:31.149 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:16:31.150 INFO:teuthology.orchestra.run.smithi005.stdout: osd map max advance = 10 2022-04-10T01:16:31.150 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:16:31.150 INFO:teuthology.orchestra.run.smithi005.stdout: journal zero on create = true 2022-04-10T01:16:31.150 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:16:31.150 INFO:teuthology.orchestra.run.smithi005.stdout: filestore ondisk finisher threads = 3 2022-04-10T01:16:31.151 INFO:teuthology.orchestra.run.smithi005.stdout: filestore apply finisher threads = 3 2022-04-10T01:16:31.151 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:16:31.151 INFO:teuthology.orchestra.run.smithi005.stdout: bdev debug aio = true 2022-04-10T01:16:31.151 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug misdirected ops = true 2022-04-10T01:16:31.152 INFO:teuthology.orchestra.run.smithi005.stdout: bluefs allocator = hybrid 2022-04-10T01:16:31.152 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore allocator = hybrid 2022-04-10T01:16:31.152 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore block size = 96636764160 2022-04-10T01:16:31.152 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore fsck on mount = True 2022-04-10T01:16:31.153 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluefs = 1/20 2022-04-10T01:16:31.153 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluestore = 1/20 2022-04-10T01:16:31.153 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2022-04-10T01:16:31.153 INFO:teuthology.orchestra.run.smithi005.stdout: debug osd = 20 2022-04-10T01:16:31.153 INFO:teuthology.orchestra.run.smithi005.stdout: debug rocksdb = 4/10 2022-04-10T01:16:31.154 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd backfillfull_ratio = 0.85 2022-04-10T01:16:31.154 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd full ratio = 0.9 2022-04-10T01:16:31.154 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd nearfull ratio = 0.8 2022-04-10T01:16:31.154 INFO:teuthology.orchestra.run.smithi005.stdout: osd failsafe full ratio = 0.95 2022-04-10T01:16:31.155 INFO:teuthology.orchestra.run.smithi005.stdout: osd objectstore = bluestore 2022-04-10T01:16:31.155 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:16:31.155 INFO:teuthology.orchestra.run.smithi005.stdout:[mgr] 2022-04-10T01:16:31.155 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2022-04-10T01:16:31.155 INFO:teuthology.orchestra.run.smithi005.stdout: debug mgr = 20 2022-04-10T01:16:31.156 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2022-04-10T01:16:31.156 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2022-04-10T01:16:31.156 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2022-04-10T01:16:31.156 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2022-04-10T01:16:31.156 INFO:teuthology.orchestra.run.smithi005.stdout: mgr/telemetry/nag = false 2022-04-10T01:16:31.157 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:16:31.157 INFO:teuthology.orchestra.run.smithi005.stdout:[mon] 2022-04-10T01:16:31.157 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2022-04-10T01:16:31.157 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2022-04-10T01:16:31.157 INFO:teuthology.orchestra.run.smithi005.stdout: debug paxos = 20 2022-04-10T01:16:31.158 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2022-04-10T01:16:31.158 INFO:teuthology.orchestra.run.smithi005.stdout: mon data avail warn = 5 2022-04-10T01:16:31.158 INFO:teuthology.orchestra.run.smithi005.stdout: mon mgr mkfs grace = 240 2022-04-10T01:16:31.158 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2022-04-10T01:16:31.158 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd reporter subtree level = osd 2022-04-10T01:16:31.159 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd prime pg temp = true 2022-04-10T01:16:31.159 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2022-04-10T01:16:31.159 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:16:31.159 INFO:teuthology.orchestra.run.smithi005.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-04-10T01:16:31.160 INFO:teuthology.orchestra.run.smithi005.stdout: auth mon ticket ttl = 660 # 11m 2022-04-10T01:16:31.160 INFO:teuthology.orchestra.run.smithi005.stdout: auth service ticket ttl = 240 # 4m 2022-04-10T01:16:31.160 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:16:31.160 INFO:teuthology.orchestra.run.smithi005.stdout: # don't complain about insecure global_id in the test suite 2022-04-10T01:16:31.160 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-04-10T01:16:31.161 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-04-10T01:16:31.161 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:16:31.161 INFO:teuthology.orchestra.run.smithi005.stdout: # 1m isn't quite enough 2022-04-10T01:16:31.161 INFO:teuthology.orchestra.run.smithi005.stdout: mon_down_mkfs_grace = 2m 2022-04-10T01:16:31.162 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:16:31.162 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_filestore_osds = false 2022-04-10T01:16:31.162 INFO:teuthology.orchestra.run.smithi005.stdout: 2022-04-10T01:16:31.162 INFO:teuthology.orchestra.run.smithi005.stdout:[client] 2022-04-10T01:16:31.162 INFO:teuthology.orchestra.run.smithi005.stdout: rgw cache enabled = true 2022-04-10T01:16:31.163 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable ops log = true 2022-04-10T01:16:31.163 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable usage log = true 2022-04-10T01:16:31.163 INFO:teuthology.orchestra.run.smithi005.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-04-10T01:16:31.163 INFO:teuthology.orchestra.run.smithi005.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-04-10T01:16:31.163 INFO:teuthology.orchestra.run.smithi005.stdout: debug rbd = 20 2022-04-10T01:16:31.164 INFO:teuthology.orchestra.run.smithi005.stdout: debug rbd_mirror = 15 2022-04-10T01:16:31.164 INFO:teuthology.orchestra.run.smithi005.stdout: log to stderr = False 2022-04-10T01:16:31.164 INFO:teuthology.orchestra.run.smithi005.stdout: rbd default features = 125 2022-04-10T01:16:31.165 INFO:teuthology.orchestra.run.smithi005.stdout: rbd mirror image policy type = simple 2022-04-10T01:16:31.165 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.a] 2022-04-10T01:16:31.165 INFO:teuthology.orchestra.run.smithi005.stdout:[client.mirror.0] 2022-04-10T01:16:31.165 INFO:teuthology.orchestra.run.smithi005.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2022-04-10T01:16:31.165 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2022-04-10T01:16:31.166 INFO:teuthology.orchestra.run.smithi005.stdout:[client.mirror.1] 2022-04-10T01:16:31.166 INFO:teuthology.orchestra.run.smithi005.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2022-04-10T01:16:31.166 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2022-04-10T01:16:31.166 INFO:teuthology.orchestra.run.smithi005.stdout:[client.mirror.2] 2022-04-10T01:16:31.166 INFO:teuthology.orchestra.run.smithi005.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2022-04-10T01:16:31.167 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2022-04-10T01:16:31.167 INFO:teuthology.orchestra.run.smithi005.stdout:[client.mirror.3] 2022-04-10T01:16:31.167 INFO:teuthology.orchestra.run.smithi005.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2022-04-10T01:16:31.167 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2022-04-10T01:16:31.168 INFO:teuthology.orchestra.run.smithi043.stdout:[global] 2022-04-10T01:16:31.168 INFO:teuthology.orchestra.run.smithi043.stdout: chdir = "" 2022-04-10T01:16:31.168 INFO:teuthology.orchestra.run.smithi043.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-04-10T01:16:31.169 INFO:teuthology.orchestra.run.smithi043.stdout: auth supported = cephx 2022-04-10T01:16:31.169 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:16:31.169 INFO:teuthology.orchestra.run.smithi043.stdout: filestore xattr use omap = true 2022-04-10T01:16:31.169 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:16:31.170 INFO:teuthology.orchestra.run.smithi043.stdout: mon clock drift allowed = 1.000 2022-04-10T01:16:31.170 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:16:31.170 INFO:teuthology.orchestra.run.smithi043.stdout: osd crush chooseleaf type = 0 2022-04-10T01:16:31.170 INFO:teuthology.orchestra.run.smithi043.stdout: auth debug = true 2022-04-10T01:16:31.170 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:16:31.171 INFO:teuthology.orchestra.run.smithi043.stdout: ms die on old message = true 2022-04-10T01:16:31.171 INFO:teuthology.orchestra.run.smithi043.stdout: ms die on bug = true 2022-04-10T01:16:31.171 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:16:31.171 INFO:teuthology.orchestra.run.smithi043.stdout: mon max pg per osd = 10000 # >= luminous 2022-04-10T01:16:31.171 INFO:teuthology.orchestra.run.smithi043.stdout: mon pg warn max object skew = 0 2022-04-10T01:16:31.172 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:16:31.172 INFO:teuthology.orchestra.run.smithi043.stdout: # disable pg_autoscaler by default for new pools 2022-04-10T01:16:31.172 INFO:teuthology.orchestra.run.smithi043.stdout: osd_pool_default_pg_autoscale_mode = off 2022-04-10T01:16:31.172 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:16:31.173 INFO:teuthology.orchestra.run.smithi043.stdout: osd pool default size = 2 2022-04-10T01:16:31.173 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:16:31.173 INFO:teuthology.orchestra.run.smithi043.stdout: mon osd allow primary affinity = true 2022-04-10T01:16:31.173 INFO:teuthology.orchestra.run.smithi043.stdout: mon osd allow pg remap = true 2022-04-10T01:16:31.173 INFO:teuthology.orchestra.run.smithi043.stdout: mon warn on legacy crush tunables = false 2022-04-10T01:16:31.174 INFO:teuthology.orchestra.run.smithi043.stdout: mon warn on crush straw calc version zero = false 2022-04-10T01:16:31.174 INFO:teuthology.orchestra.run.smithi043.stdout: mon warn on no sortbitwise = false 2022-04-10T01:16:31.174 INFO:teuthology.orchestra.run.smithi043.stdout: mon warn on osd down out interval zero = false 2022-04-10T01:16:31.174 INFO:teuthology.orchestra.run.smithi043.stdout: mon warn on too few osds = false 2022-04-10T01:16:31.174 INFO:teuthology.orchestra.run.smithi043.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-04-10T01:16:31.175 INFO:teuthology.orchestra.run.smithi043.stdout: mon_warn_on_pool_no_redundancy = false 2022-04-10T01:16:31.175 INFO:teuthology.orchestra.run.smithi043.stdout: mon_allow_pool_size_one = true 2022-04-10T01:16:31.175 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:16:31.175 INFO:teuthology.orchestra.run.smithi043.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2022-04-10T01:16:31.176 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:16:31.176 INFO:teuthology.orchestra.run.smithi043.stdout: osd default data pool replay window = 5 2022-04-10T01:16:31.176 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:16:31.176 INFO:teuthology.orchestra.run.smithi043.stdout: mon allow pool delete = true 2022-04-10T01:16:31.176 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:16:31.176 INFO:teuthology.orchestra.run.smithi043.stdout: mon cluster log file level = debug 2022-04-10T01:16:31.177 INFO:teuthology.orchestra.run.smithi043.stdout: debug asserts on shutdown = true 2022-04-10T01:16:31.177 INFO:teuthology.orchestra.run.smithi043.stdout: mon health detail to clog = false 2022-04-10T01:16:31.177 INFO:teuthology.orchestra.run.smithi043.stdout: mon host = 172.21.15.43 2022-04-10T01:16:31.177 INFO:teuthology.orchestra.run.smithi043.stdout: mon client directed command retry = 5 2022-04-10T01:16:31.178 INFO:teuthology.orchestra.run.smithi043.stdout: ms inject socket failures = 5000 2022-04-10T01:16:31.178 INFO:teuthology.orchestra.run.smithi043.stdout: fsid = 1f0791d0-116c-427c-a4dc-de372595f89c 2022-04-10T01:16:31.178 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:16:31.178 INFO:teuthology.orchestra.run.smithi043.stdout:[osd] 2022-04-10T01:16:31.178 INFO:teuthology.orchestra.run.smithi043.stdout: osd journal size = 100 2022-04-10T01:16:31.179 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:16:31.179 INFO:teuthology.orchestra.run.smithi043.stdout: osd scrub load threshold = 5.0 2022-04-10T01:16:31.179 INFO:teuthology.orchestra.run.smithi043.stdout: osd scrub max interval = 600 2022-04-10T01:16:31.179 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:16:31.179 INFO:teuthology.orchestra.run.smithi043.stdout: osd recover clone overlap = true 2022-04-10T01:16:31.180 INFO:teuthology.orchestra.run.smithi043.stdout: osd recovery max chunk = 1048576 2022-04-10T01:16:31.180 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:16:31.180 INFO:teuthology.orchestra.run.smithi043.stdout: osd debug shutdown = true 2022-04-10T01:16:31.180 INFO:teuthology.orchestra.run.smithi043.stdout: osd debug op order = true 2022-04-10T01:16:31.180 INFO:teuthology.orchestra.run.smithi043.stdout: osd debug verify stray on activate = true 2022-04-10T01:16:31.181 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:16:31.181 INFO:teuthology.orchestra.run.smithi043.stdout: osd open classes on start = true 2022-04-10T01:16:31.181 INFO:teuthology.orchestra.run.smithi043.stdout: osd debug pg log writeout = true 2022-04-10T01:16:31.181 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:16:31.182 INFO:teuthology.orchestra.run.smithi043.stdout: osd deep scrub update digest min age = 30 2022-04-10T01:16:31.182 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:16:31.182 INFO:teuthology.orchestra.run.smithi043.stdout: osd map max advance = 10 2022-04-10T01:16:31.182 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:16:31.182 INFO:teuthology.orchestra.run.smithi043.stdout: journal zero on create = true 2022-04-10T01:16:31.183 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:16:31.183 INFO:teuthology.orchestra.run.smithi043.stdout: filestore ondisk finisher threads = 3 2022-04-10T01:16:31.183 INFO:teuthology.orchestra.run.smithi043.stdout: filestore apply finisher threads = 3 2022-04-10T01:16:31.183 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:16:31.183 INFO:teuthology.orchestra.run.smithi043.stdout: bdev debug aio = true 2022-04-10T01:16:31.184 INFO:teuthology.orchestra.run.smithi043.stdout: osd debug misdirected ops = true 2022-04-10T01:16:31.184 INFO:teuthology.orchestra.run.smithi043.stdout: bluefs allocator = hybrid 2022-04-10T01:16:31.184 INFO:teuthology.orchestra.run.smithi043.stdout: bluestore allocator = hybrid 2022-04-10T01:16:31.184 INFO:teuthology.orchestra.run.smithi043.stdout: bluestore block size = 96636764160 2022-04-10T01:16:31.184 INFO:teuthology.orchestra.run.smithi043.stdout: bluestore fsck on mount = True 2022-04-10T01:16:31.185 INFO:teuthology.orchestra.run.smithi043.stdout: debug bluefs = 1/20 2022-04-10T01:16:31.185 INFO:teuthology.orchestra.run.smithi043.stdout: debug bluestore = 1/20 2022-04-10T01:16:31.185 INFO:teuthology.orchestra.run.smithi043.stdout: debug ms = 1 2022-04-10T01:16:31.185 INFO:teuthology.orchestra.run.smithi043.stdout: debug osd = 20 2022-04-10T01:16:31.185 INFO:teuthology.orchestra.run.smithi043.stdout: debug rocksdb = 4/10 2022-04-10T01:16:31.186 INFO:teuthology.orchestra.run.smithi043.stdout: mon osd backfillfull_ratio = 0.85 2022-04-10T01:16:31.186 INFO:teuthology.orchestra.run.smithi043.stdout: mon osd full ratio = 0.9 2022-04-10T01:16:31.186 INFO:teuthology.orchestra.run.smithi043.stdout: mon osd nearfull ratio = 0.8 2022-04-10T01:16:31.186 INFO:teuthology.orchestra.run.smithi043.stdout: osd failsafe full ratio = 0.95 2022-04-10T01:16:31.187 INFO:teuthology.orchestra.run.smithi043.stdout: osd objectstore = bluestore 2022-04-10T01:16:31.187 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:16:31.187 INFO:teuthology.orchestra.run.smithi043.stdout:[mgr] 2022-04-10T01:16:31.187 INFO:teuthology.orchestra.run.smithi043.stdout: debug ms = 1 2022-04-10T01:16:31.187 INFO:teuthology.orchestra.run.smithi043.stdout: debug mgr = 20 2022-04-10T01:16:31.188 INFO:teuthology.orchestra.run.smithi043.stdout: debug mon = 20 2022-04-10T01:16:31.188 INFO:teuthology.orchestra.run.smithi043.stdout: debug auth = 20 2022-04-10T01:16:31.188 INFO:teuthology.orchestra.run.smithi043.stdout: mon reweight min pgs per osd = 4 2022-04-10T01:16:31.188 INFO:teuthology.orchestra.run.smithi043.stdout: mon reweight min bytes per osd = 10 2022-04-10T01:16:31.188 INFO:teuthology.orchestra.run.smithi043.stdout: mgr/telemetry/nag = false 2022-04-10T01:16:31.189 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:16:31.189 INFO:teuthology.orchestra.run.smithi043.stdout:[mon] 2022-04-10T01:16:31.189 INFO:teuthology.orchestra.run.smithi043.stdout: debug ms = 1 2022-04-10T01:16:31.189 INFO:teuthology.orchestra.run.smithi043.stdout: debug mon = 20 2022-04-10T01:16:31.189 INFO:teuthology.orchestra.run.smithi043.stdout: debug paxos = 20 2022-04-10T01:16:31.190 INFO:teuthology.orchestra.run.smithi043.stdout: debug auth = 20 2022-04-10T01:16:31.190 INFO:teuthology.orchestra.run.smithi043.stdout: mon data avail warn = 5 2022-04-10T01:16:31.190 INFO:teuthology.orchestra.run.smithi043.stdout: mon mgr mkfs grace = 240 2022-04-10T01:16:31.190 INFO:teuthology.orchestra.run.smithi043.stdout: mon reweight min pgs per osd = 4 2022-04-10T01:16:31.190 INFO:teuthology.orchestra.run.smithi043.stdout: mon osd reporter subtree level = osd 2022-04-10T01:16:31.191 INFO:teuthology.orchestra.run.smithi043.stdout: mon osd prime pg temp = true 2022-04-10T01:16:31.191 INFO:teuthology.orchestra.run.smithi043.stdout: mon reweight min bytes per osd = 10 2022-04-10T01:16:31.191 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:16:31.191 INFO:teuthology.orchestra.run.smithi043.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-04-10T01:16:31.191 INFO:teuthology.orchestra.run.smithi043.stdout: auth mon ticket ttl = 660 # 11m 2022-04-10T01:16:31.192 INFO:teuthology.orchestra.run.smithi043.stdout: auth service ticket ttl = 240 # 4m 2022-04-10T01:16:31.192 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:16:31.192 INFO:teuthology.orchestra.run.smithi043.stdout: # don't complain about insecure global_id in the test suite 2022-04-10T01:16:31.192 INFO:teuthology.orchestra.run.smithi043.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-04-10T01:16:31.192 INFO:teuthology.orchestra.run.smithi043.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-04-10T01:16:31.193 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:16:31.193 INFO:teuthology.orchestra.run.smithi043.stdout: # 1m isn't quite enough 2022-04-10T01:16:31.193 INFO:teuthology.orchestra.run.smithi043.stdout: mon_down_mkfs_grace = 2m 2022-04-10T01:16:31.193 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:16:31.193 INFO:teuthology.orchestra.run.smithi043.stdout: mon_warn_on_filestore_osds = false 2022-04-10T01:16:31.194 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:16:31.194 INFO:teuthology.orchestra.run.smithi043.stdout:[client] 2022-04-10T01:16:31.194 INFO:teuthology.orchestra.run.smithi043.stdout: rgw cache enabled = true 2022-04-10T01:16:31.194 INFO:teuthology.orchestra.run.smithi043.stdout: rgw enable ops log = true 2022-04-10T01:16:31.194 INFO:teuthology.orchestra.run.smithi043.stdout: rgw enable usage log = true 2022-04-10T01:16:31.195 INFO:teuthology.orchestra.run.smithi043.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-04-10T01:16:31.195 INFO:teuthology.orchestra.run.smithi043.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-04-10T01:16:31.195 INFO:teuthology.orchestra.run.smithi043.stdout: debug rbd = 20 2022-04-10T01:16:31.195 INFO:teuthology.orchestra.run.smithi043.stdout: debug rbd_mirror = 15 2022-04-10T01:16:31.195 INFO:teuthology.orchestra.run.smithi043.stdout: log to stderr = False 2022-04-10T01:16:31.196 INFO:teuthology.orchestra.run.smithi043.stdout: rbd default features = 125 2022-04-10T01:16:31.196 INFO:teuthology.orchestra.run.smithi043.stdout: rbd mirror image policy type = simple 2022-04-10T01:16:31.196 INFO:teuthology.orchestra.run.smithi043.stdout:[mon.a] 2022-04-10T01:16:31.196 INFO:teuthology.orchestra.run.smithi043.stdout:[client.mirror.0] 2022-04-10T01:16:31.196 INFO:teuthology.orchestra.run.smithi043.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2022-04-10T01:16:31.197 INFO:teuthology.orchestra.run.smithi043.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2022-04-10T01:16:31.197 INFO:teuthology.orchestra.run.smithi043.stdout:[client.mirror.1] 2022-04-10T01:16:31.197 INFO:teuthology.orchestra.run.smithi043.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2022-04-10T01:16:31.197 INFO:teuthology.orchestra.run.smithi043.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2022-04-10T01:16:31.197 INFO:teuthology.orchestra.run.smithi043.stdout:[client.mirror.2] 2022-04-10T01:16:31.198 INFO:teuthology.orchestra.run.smithi043.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2022-04-10T01:16:31.198 INFO:teuthology.orchestra.run.smithi043.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2022-04-10T01:16:31.198 INFO:teuthology.orchestra.run.smithi043.stdout:[client.mirror.3] 2022-04-10T01:16:31.198 INFO:teuthology.orchestra.run.smithi043.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2022-04-10T01:16:31.198 INFO:teuthology.orchestra.run.smithi043.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2022-04-10T01:16:31.200 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2022-04-10T01:16:31.200 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2022-04-10T01:16:31.254 INFO:tasks.ceph:Copying monmap to all nodes... 2022-04-10T01:16:31.254 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-04-10T01:16:31.255 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2022-04-10T01:16:31.272 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-04-10T01:16:31.272 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2022-04-10T01:16:31.329 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi005.front.sepia.ceph.com 2022-04-10T01:16:31.330 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-04-10T01:16:31.330 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/ceph/cluster2.keyring 2022-04-10T01:16:31.330 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2022-04-10T01:16:31.373 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-04-10T01:16:31.373 DEBUG:teuthology.orchestra.run.smithi005:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2022-04-10T01:16:31.431 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi043.front.sepia.ceph.com 2022-04-10T01:16:31.431 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-04-10T01:16:31.432 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/ceph/cluster2.keyring 2022-04-10T01:16:31.432 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2022-04-10T01:16:31.469 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-04-10T01:16:31.469 DEBUG:teuthology.orchestra.run.smithi043:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2022-04-10T01:16:31.526 INFO:tasks.ceph:Setting up mon nodes... 2022-04-10T01:16:31.526 INFO:tasks.ceph:Setting up mgr nodes... 2022-04-10T01:16:31.527 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2022-04-10T01:16:31.581 INFO:teuthology.orchestra.run.smithi005.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2022-04-10T01:16:31.584 INFO:tasks.ceph:Setting up mds nodes... 2022-04-10T01:16:31.584 INFO:tasks.ceph_client:Setting up client nodes... 2022-04-10T01:16:31.585 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2022-04-10T01:16:31.626 INFO:teuthology.orchestra.run.smithi005.stdout:creating /etc/ceph/cluster2.client.0.keyring 2022-04-10T01:16:31.640 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2022-04-10T01:16:31.680 INFO:teuthology.orchestra.run.smithi043.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2022-04-10T01:16:31.693 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2022-04-10T01:16:31.773 INFO:teuthology.orchestra.run.smithi043.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2022-04-10T01:16:31.786 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2022-04-10T01:16:31.866 INFO:teuthology.orchestra.run.smithi043.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2022-04-10T01:16:31.879 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2022-04-10T01:16:31.960 INFO:teuthology.orchestra.run.smithi043.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2022-04-10T01:16:31.973 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2022-04-10T01:16:32.054 INFO:teuthology.orchestra.run.smithi043.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2022-04-10T01:16:32.067 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2022-04-10T01:16:32.147 INFO:teuthology.orchestra.run.smithi043.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2022-04-10T01:16:32.160 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2022-04-10T01:16:32.241 INFO:teuthology.orchestra.run.smithi043.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2022-04-10T01:16:32.254 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2022-04-10T01:16:32.334 INFO:teuthology.orchestra.run.smithi043.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2022-04-10T01:16:32.347 INFO:tasks.ceph:Running mkfs on osd nodes... 2022-04-10T01:16:32.348 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi005.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi043.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2022-04-10T01:16:32.348 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2022-04-10T01:16:32.414 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2022-04-10T01:16:32.414 INFO:tasks.ceph:role: cluster2.osd.0 2022-04-10T01:16:32.415 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi043.front.sepia.ceph.com 2022-04-10T01:16:32.415 DEBUG:teuthology.orchestra.run.smithi043:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2022-04-10T01:16:32.480 INFO:teuthology.orchestra.run.smithi043.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2022-04-10T01:16:32.480 INFO:teuthology.orchestra.run.smithi043.stdout: = sectsz=512 attr=2, projid32bit=1 2022-04-10T01:16:32.481 INFO:teuthology.orchestra.run.smithi043.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-04-10T01:16:32.481 INFO:teuthology.orchestra.run.smithi043.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-04-10T01:16:32.481 INFO:teuthology.orchestra.run.smithi043.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-04-10T01:16:32.481 INFO:teuthology.orchestra.run.smithi043.stdout: = sunit=0 swidth=0 blks 2022-04-10T01:16:32.481 INFO:teuthology.orchestra.run.smithi043.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-04-10T01:16:32.482 INFO:teuthology.orchestra.run.smithi043.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-04-10T01:16:32.482 INFO:teuthology.orchestra.run.smithi043.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-04-10T01:16:32.482 INFO:teuthology.orchestra.run.smithi043.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-04-10T01:16:32.503 INFO:teuthology.orchestra.run.smithi043.stdout:Discarding blocks...Done. 2022-04-10T01:16:32.505 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi043.front.sepia.ceph.com -o noatime 2022-04-10T01:16:32.505 DEBUG:teuthology.orchestra.run.smithi043:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2022-04-10T01:16:32.585 DEBUG:teuthology.orchestra.run.smithi043:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2022-04-10T01:16:32.657 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2022-04-10T01:16:32.724 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2022-04-10T01:16:32.724 INFO:tasks.ceph:role: cluster2.osd.1 2022-04-10T01:16:32.724 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi043.front.sepia.ceph.com 2022-04-10T01:16:32.725 DEBUG:teuthology.orchestra.run.smithi043:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2022-04-10T01:16:32.789 INFO:teuthology.orchestra.run.smithi043.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2022-04-10T01:16:32.790 INFO:teuthology.orchestra.run.smithi043.stdout: = sectsz=512 attr=2, projid32bit=1 2022-04-10T01:16:32.790 INFO:teuthology.orchestra.run.smithi043.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-04-10T01:16:32.790 INFO:teuthology.orchestra.run.smithi043.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-04-10T01:16:32.790 INFO:teuthology.orchestra.run.smithi043.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-04-10T01:16:32.791 INFO:teuthology.orchestra.run.smithi043.stdout: = sunit=0 swidth=0 blks 2022-04-10T01:16:32.791 INFO:teuthology.orchestra.run.smithi043.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-04-10T01:16:32.791 INFO:teuthology.orchestra.run.smithi043.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-04-10T01:16:32.791 INFO:teuthology.orchestra.run.smithi043.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-04-10T01:16:32.791 INFO:teuthology.orchestra.run.smithi043.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-04-10T01:16:32.812 INFO:teuthology.orchestra.run.smithi043.stdout:Discarding blocks...Done. 2022-04-10T01:16:32.814 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi043.front.sepia.ceph.com -o noatime 2022-04-10T01:16:32.814 DEBUG:teuthology.orchestra.run.smithi043:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2022-04-10T01:16:32.894 DEBUG:teuthology.orchestra.run.smithi043:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2022-04-10T01:16:32.966 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2022-04-10T01:16:33.033 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2022-04-10T01:16:33.033 INFO:tasks.ceph:role: cluster2.osd.2 2022-04-10T01:16:33.034 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi043.front.sepia.ceph.com 2022-04-10T01:16:33.034 DEBUG:teuthology.orchestra.run.smithi043:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2022-04-10T01:16:33.099 INFO:teuthology.orchestra.run.smithi043.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2022-04-10T01:16:33.099 INFO:teuthology.orchestra.run.smithi043.stdout: = sectsz=512 attr=2, projid32bit=1 2022-04-10T01:16:33.099 INFO:teuthology.orchestra.run.smithi043.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-04-10T01:16:33.100 INFO:teuthology.orchestra.run.smithi043.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-04-10T01:16:33.100 INFO:teuthology.orchestra.run.smithi043.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-04-10T01:16:33.100 INFO:teuthology.orchestra.run.smithi043.stdout: = sunit=0 swidth=0 blks 2022-04-10T01:16:33.100 INFO:teuthology.orchestra.run.smithi043.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-04-10T01:16:33.101 INFO:teuthology.orchestra.run.smithi043.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-04-10T01:16:33.101 INFO:teuthology.orchestra.run.smithi043.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-04-10T01:16:33.101 INFO:teuthology.orchestra.run.smithi043.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-04-10T01:16:33.122 INFO:teuthology.orchestra.run.smithi043.stdout:Discarding blocks...Done. 2022-04-10T01:16:33.124 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi043.front.sepia.ceph.com -o noatime 2022-04-10T01:16:33.124 DEBUG:teuthology.orchestra.run.smithi043:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2022-04-10T01:16:33.205 DEBUG:teuthology.orchestra.run.smithi043:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2022-04-10T01:16:33.276 DEBUG:teuthology.orchestra.run.smithi043:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2022-04-10T01:16:33.361 INFO:teuthology.orchestra.run.smithi043.stderr:2022-04-10T01:16:33.359+0000 7fc1b579e3c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2022-04-10T01:16:33.361 INFO:teuthology.orchestra.run.smithi043.stderr:2022-04-10T01:16:33.359+0000 7fc1b579e3c0 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2022-04-10T01:16:33.361 INFO:teuthology.orchestra.run.smithi043.stderr:2022-04-10T01:16:33.359+0000 7fc1b579e3c0 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2022-04-10T01:16:33.362 INFO:teuthology.orchestra.run.smithi043.stderr:2022-04-10T01:16:33.359+0000 7fc1b579e3c0 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2022-04-10T01:16:33.362 INFO:teuthology.orchestra.run.smithi043.stderr:2022-04-10T01:16:33.359+0000 7fc1b579e3c0 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2022-04-10T01:16:33.362 INFO:teuthology.orchestra.run.smithi043.stderr:2022-04-10T01:16:33.359+0000 7fc1b579e3c0 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid 2022-04-10T01:16:36.219 DEBUG:teuthology.orchestra.run.smithi043:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2022-04-10T01:16:36.296 DEBUG:teuthology.orchestra.run.smithi043:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2022-04-10T01:16:36.386 INFO:teuthology.orchestra.run.smithi043.stderr:2022-04-10T01:16:36.384+0000 7f7056e813c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2022-04-10T01:16:36.387 INFO:teuthology.orchestra.run.smithi043.stderr:2022-04-10T01:16:36.385+0000 7f7056e813c0 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2022-04-10T01:16:36.387 INFO:teuthology.orchestra.run.smithi043.stderr:2022-04-10T01:16:36.385+0000 7f7056e813c0 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2022-04-10T01:16:36.388 INFO:teuthology.orchestra.run.smithi043.stderr:2022-04-10T01:16:36.385+0000 7f7056e813c0 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2022-04-10T01:16:36.388 INFO:teuthology.orchestra.run.smithi043.stderr:2022-04-10T01:16:36.385+0000 7f7056e813c0 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2022-04-10T01:16:36.388 INFO:teuthology.orchestra.run.smithi043.stderr:2022-04-10T01:16:36.385+0000 7f7056e813c0 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid 2022-04-10T01:16:39.250 DEBUG:teuthology.orchestra.run.smithi043:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2022-04-10T01:16:39.284 DEBUG:teuthology.orchestra.run.smithi043:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2022-04-10T01:16:39.375 INFO:teuthology.orchestra.run.smithi043.stderr:2022-04-10T01:16:39.373+0000 7f3b5c53a3c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2022-04-10T01:16:39.375 INFO:teuthology.orchestra.run.smithi043.stderr:2022-04-10T01:16:39.373+0000 7f3b5c53a3c0 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2022-04-10T01:16:39.376 INFO:teuthology.orchestra.run.smithi043.stderr:2022-04-10T01:16:39.373+0000 7f3b5c53a3c0 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2022-04-10T01:16:39.376 INFO:teuthology.orchestra.run.smithi043.stderr:2022-04-10T01:16:39.373+0000 7f3b5c53a3c0 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2022-04-10T01:16:39.376 INFO:teuthology.orchestra.run.smithi043.stderr:2022-04-10T01:16:39.373+0000 7f3b5c53a3c0 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2022-04-10T01:16:39.376 INFO:teuthology.orchestra.run.smithi043.stderr:2022-04-10T01:16:39.373+0000 7f3b5c53a3c0 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid 2022-04-10T01:16:42.226 DEBUG:teuthology.orchestra.run.smithi043:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2022-04-10T01:16:42.259 INFO:tasks.ceph:Reading keys from all nodes... 2022-04-10T01:16:42.260 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-04-10T01:16:42.260 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2022-04-10T01:16:42.294 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-04-10T01:16:42.295 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2022-04-10T01:16:42.330 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-04-10T01:16:42.330 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2022-04-10T01:16:42.402 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-04-10T01:16:42.403 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2022-04-10T01:16:42.474 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2022-04-10T01:16:42.474 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2022-04-10T01:16:42.495 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-04-10T01:16:42.496 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2022-04-10T01:16:42.533 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-04-10T01:16:42.534 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2022-04-10T01:16:42.591 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-04-10T01:16:42.592 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2022-04-10T01:16:42.652 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-04-10T01:16:42.653 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2022-04-10T01:16:42.712 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-04-10T01:16:42.713 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2022-04-10T01:16:42.772 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-04-10T01:16:42.773 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2022-04-10T01:16:42.831 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-04-10T01:16:42.832 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2022-04-10T01:16:42.893 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-04-10T01:16:42.893 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2022-04-10T01:16:42.953 INFO:tasks.ceph:Adding keys to all mons... 2022-04-10T01:16:42.954 DEBUG:teuthology.orchestra.run.smithi043:> sudo tee -a /etc/ceph/cluster2.keyring 2022-04-10T01:16:43.024 INFO:teuthology.orchestra.run.smithi043.stdout:[mgr.x] 2022-04-10T01:16:43.024 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQDvL1Jia1WvIhAAb+DJaSyE74yQfKsp2A89Fg== 2022-04-10T01:16:43.024 INFO:teuthology.orchestra.run.smithi043.stdout:[osd.0] 2022-04-10T01:16:43.025 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQDxL1JiNzB8FRAAsVPM4gaQmLk2I9uZtMic/g== 2022-04-10T01:16:43.025 INFO:teuthology.orchestra.run.smithi043.stdout:[osd.1] 2022-04-10T01:16:43.026 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQD0L1JiawQDFxAAh1r8GjHfYfy8kK8WFTMFVw== 2022-04-10T01:16:43.026 INFO:teuthology.orchestra.run.smithi043.stdout:[osd.2] 2022-04-10T01:16:43.026 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQD3L1Jix0xVFhAANgli3NLLwiTpDGkGrIX84A== 2022-04-10T01:16:43.026 INFO:teuthology.orchestra.run.smithi043.stdout:[client.0] 2022-04-10T01:16:43.027 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQDvL1JiVKdcJRAAlhKdKKhI4xwEJqj/9wTxFA== 2022-04-10T01:16:43.027 INFO:teuthology.orchestra.run.smithi043.stdout:[client.mirror] 2022-04-10T01:16:43.027 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQDvL1Ji52OOKBAA/f28IwoQIDPId5p8KR0QnQ== 2022-04-10T01:16:43.028 INFO:teuthology.orchestra.run.smithi043.stdout:[client.mirror.0] 2022-04-10T01:16:43.028 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQDvL1JiIDMfLhAAJ77wIKLWjqdvsir8sTv+JQ== 2022-04-10T01:16:43.029 INFO:teuthology.orchestra.run.smithi043.stdout:[client.mirror.1] 2022-04-10T01:16:43.029 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQDvL1Jig9WoMxAA/CdnJcPIs47w1kY6RtVFSQ== 2022-04-10T01:16:43.029 INFO:teuthology.orchestra.run.smithi043.stdout:[client.mirror.2] 2022-04-10T01:16:43.030 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQDvL1Jiy7A5ORAA4AQH7kNoQ36b6T2DGN4hbA== 2022-04-10T01:16:43.030 INFO:teuthology.orchestra.run.smithi043.stdout:[client.mirror.3] 2022-04-10T01:16:43.030 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQDwL1Ji1Tk+AxAAi661CPW7+jMCaIg+YSfYxQ== 2022-04-10T01:16:43.031 INFO:teuthology.orchestra.run.smithi043.stdout:[client.mirror.4] 2022-04-10T01:16:43.031 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQDwL1JiVhbQCBAAOvlEU/tqkL5jkYgJ1GSAQA== 2022-04-10T01:16:43.031 INFO:teuthology.orchestra.run.smithi043.stdout:[client.mirror.5] 2022-04-10T01:16:43.031 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQDwL1JiuRtiDhAA0AoyaGE69MgJ+EYQfhUcFA== 2022-04-10T01:16:43.032 INFO:teuthology.orchestra.run.smithi043.stdout:[client.mirror.6] 2022-04-10T01:16:43.032 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQDwL1JiYaf2ExAAVyrR1Dy3WIaWrXh9N506Pw== 2022-04-10T01:16:43.033 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2022-04-10T01:16:43.128 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-04-10T01:16:43.181 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-04-10T01:16:43.232 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-04-10T01:16:43.283 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-04-10T01:16:43.337 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-04-10T01:16:43.390 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-04-10T01:16:43.443 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-04-10T01:16:43.496 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-04-10T01:16:43.550 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-04-10T01:16:43.604 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-04-10T01:16:43.657 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-04-10T01:16:43.710 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-04-10T01:16:43.764 INFO:tasks.ceph:Running mkfs on mon nodes... 2022-04-10T01:16:43.764 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2022-04-10T01:16:43.791 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2022-04-10T01:16:43.885 DEBUG:teuthology.orchestra.run.smithi043:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2022-04-10T01:16:43.912 DEBUG:teuthology.orchestra.run.smithi043:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2022-04-10T01:16:43.968 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2022-04-10T01:16:43.969 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2022-04-10T01:16:43.969 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2022-04-10T01:16:44.011 INFO:tasks.ceph.cluster2.mon.a:Started 2022-04-10T01:16:44.012 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2022-04-10T01:16:44.012 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2022-04-10T01:16:44.012 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2022-04-10T01:16:44.015 INFO:tasks.ceph.cluster2.mgr.x:Started 2022-04-10T01:16:44.016 INFO:tasks.ceph:Setting crush tunables to default 2022-04-10T01:16:44.016 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph --cluster cluster2 osd crush tunables default 2022-04-10T01:16:44.220 INFO:teuthology.orchestra.run.smithi043.stderr:adjusted tunables profile to default 2022-04-10T01:16:44.231 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2022-04-10T01:16:44.232 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-04-10T01:16:44.232 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2022-04-10T01:16:44.259 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-04-10T01:16:44.260 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2022-04-10T01:16:44.304 INFO:tasks.ceph.cluster2.mgr.x.smithi005.stderr:2022-04-10T01:16:44.303+0000 7f76030f4000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-04-10T01:16:44.325 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-04-10T01:16:44.326 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2022-04-10T01:16:44.376 INFO:tasks.ceph.cluster2.mgr.x.smithi005.stderr:2022-04-10T01:16:44.376+0000 7f76030f4000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-04-10T01:16:44.392 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph --cluster cluster2 osd new debf21d4-4a03-4599-9cb6-c88f44285145 0 2022-04-10T01:16:44.455 INFO:tasks.ceph.cluster2.mgr.x.smithi005.stderr:2022-04-10T01:16:44.454+0000 7f76030f4000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-04-10T01:16:44.523 INFO:tasks.ceph.cluster2.mgr.x.smithi005.stderr:2022-04-10T01:16:44.522+0000 7f76030f4000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-04-10T01:16:44.590 INFO:tasks.ceph.cluster2.mgr.x.smithi005.stderr:2022-04-10T01:16:44.589+0000 7f76030f4000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-04-10T01:16:44.641 INFO:teuthology.orchestra.run.smithi043.stdout:0 2022-04-10T01:16:44.654 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph --cluster cluster2 osd new 28e8cb43-d9b1-4e2f-8639-138aa9881798 1 2022-04-10T01:16:44.758 INFO:tasks.ceph.cluster2.mgr.x.smithi005.stderr:2022-04-10T01:16:44.757+0000 7f76030f4000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-04-10T01:16:44.827 INFO:tasks.ceph.cluster2.mgr.x.smithi005.stderr:2022-04-10T01:16:44.826+0000 7f76030f4000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-04-10T01:16:44.864 INFO:teuthology.orchestra.run.smithi043.stdout:1 2022-04-10T01:16:44.876 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph --cluster cluster2 osd new 333de03e-cd2c-4d16-9535-027c7d0bac5e 2 2022-04-10T01:16:44.907 INFO:tasks.ceph.cluster2.mgr.x.smithi005.stderr:2022-04-10T01:16:44.906+0000 7f76030f4000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-04-10T01:16:45.087 INFO:teuthology.orchestra.run.smithi043.stdout:2 2022-04-10T01:16:45.099 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2022-04-10T01:16:45.099 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2022-04-10T01:16:45.102 INFO:tasks.ceph.cluster2.osd.0:Started 2022-04-10T01:16:45.102 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2022-04-10T01:16:45.102 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2022-04-10T01:16:45.104 INFO:tasks.ceph.cluster2.osd.1:Started 2022-04-10T01:16:45.105 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2022-04-10T01:16:45.105 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2022-04-10T01:16:45.107 INFO:tasks.ceph.cluster2.osd.2:Started 2022-04-10T01:16:45.108 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2022-04-10T01:16:45.157 INFO:tasks.ceph.cluster2.mgr.x.smithi005.stderr:2022-04-10T01:16:45.156+0000 7f76030f4000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-04-10T01:16:45.242 INFO:tasks.ceph.cluster2.mgr.x.smithi005.stderr:2022-04-10T01:16:45.241+0000 7f76030f4000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-04-10T01:16:45.314 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:16:45.315 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":5,"fsid":"1f0791d0-116c-427c-a4dc-de372595f89c","created":"2022-04-10T01:16:44.122952+0000","modified":"2022-04-10T01:16:45.085333+0000","last_up_change":"0.000000","last_in_change":"2022-04-10T01:16:45.085333+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"debf21d4-4a03-4599-9cb6-c88f44285145","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"28e8cb43-d9b1-4e2f-8639-138aa9881798","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"333de03e-cd2c-4d16-9535-027c7d0bac5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-04-10T01:16:45.327 INFO:tasks.ceph.ceph_manager.cluster2:[] 2022-04-10T01:16:45.327 INFO:tasks.ceph:Waiting for OSDs to come up 2022-04-10T01:16:45.371 INFO:tasks.ceph.cluster2.mgr.x.smithi005.stderr:2022-04-10T01:16:45.370+0000 7f76030f4000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-04-10T01:16:45.630 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2022-04-10T01:16:45.700 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr:2022-04-10T01:16:45.698+0000 7f5316d4a3c0 -1 Falling back to public interface 2022-04-10T01:16:45.703 INFO:tasks.ceph.cluster2.osd.1.smithi043.stderr:2022-04-10T01:16:45.702+0000 7f181b6423c0 -1 Falling back to public interface 2022-04-10T01:16:45.704 INFO:tasks.ceph.cluster2.osd.2.smithi043.stderr:2022-04-10T01:16:45.703+0000 7f8bfa8c73c0 -1 Falling back to public interface 2022-04-10T01:16:45.705 INFO:tasks.ceph.cluster2.osd.1.smithi043.stderr:2022-04-10T01:16:45.703+0000 7f1815a64700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6812/0: (98) Address already in use 2022-04-10T01:16:45.820 INFO:teuthology.misc.health.smithi043.stdout: 2022-04-10T01:16:45.820 INFO:teuthology.misc.health.smithi043.stdout:{"epoch":5,"fsid":"1f0791d0-116c-427c-a4dc-de372595f89c","created":"2022-04-10T01:16:44.122952+0000","modified":"2022-04-10T01:16:45.085333+0000","last_up_change":"0.000000","last_in_change":"2022-04-10T01:16:45.085333+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"debf21d4-4a03-4599-9cb6-c88f44285145","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"28e8cb43-d9b1-4e2f-8639-138aa9881798","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"333de03e-cd2c-4d16-9535-027c7d0bac5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-04-10T01:16:45.831 DEBUG:teuthology.misc:0 of 3 OSDs are up 2022-04-10T01:16:45.886 INFO:tasks.ceph.cluster2.mgr.x.smithi005.stderr:2022-04-10T01:16:45.885+0000 7f76030f4000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-04-10T01:16:45.961 INFO:tasks.ceph.cluster2.mgr.x.smithi005.stderr:2022-04-10T01:16:45.960+0000 7f76030f4000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-04-10T01:16:46.095 INFO:tasks.ceph.cluster2.mgr.x.smithi005.stderr:2022-04-10T01:16:46.094+0000 7f76030f4000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-04-10T01:16:46.324 INFO:tasks.ceph.cluster2.mgr.x.smithi005.stderr:2022-04-10T01:16:46.323+0000 7f76030f4000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-04-10T01:16:47.030 INFO:tasks.ceph.cluster2.mgr.x.smithi005.stderr:2022-04-10T01:16:47.029+0000 7f76030f4000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-04-10T01:16:47.173 INFO:tasks.ceph.cluster2.mgr.x.smithi005.stderr:2022-04-10T01:16:47.172+0000 7f76030f4000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-04-10T01:16:47.530 INFO:tasks.ceph.cluster2.mgr.x.smithi005.stderr:2022-04-10T01:16:47.529+0000 7f76030f4000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-04-10T01:16:47.806 INFO:tasks.ceph.cluster2.mgr.x.smithi005.stderr:2022-04-10T01:16:47.805+0000 7f76030f4000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-04-10T01:16:47.886 INFO:tasks.ceph.cluster2.mgr.x.smithi005.stderr:2022-04-10T01:16:47.885+0000 7f76030f4000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-04-10T01:16:47.958 INFO:tasks.ceph.cluster2.mgr.x.smithi005.stderr:2022-04-10T01:16:47.956+0000 7f76030f4000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-04-10T01:16:48.031 INFO:tasks.ceph.cluster2.mgr.x.smithi005.stderr:2022-04-10T01:16:48.030+0000 7f76030f4000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-04-10T01:16:48.304 INFO:tasks.ceph.cluster2.mgr.x.smithi005.stderr:2022-04-10T01:16:48.303+0000 7f76030f4000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-04-10T01:16:49.026 INFO:tasks.ceph.cluster2.mgr.x.smithi005.stderr:2022-04-10T01:16:49.025+0000 7f76030f4000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-04-10T01:16:49.166 INFO:tasks.ceph.cluster2.osd.2.smithi043.stderr:2022-04-10T01:16:49.164+0000 7f8bfa8c73c0 -1 osd.2 0 log_to_monitors true 2022-04-10T01:16:49.217 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr:2022-04-10T01:16:49.216+0000 7f5316d4a3c0 -1 osd.0 0 log_to_monitors true 2022-04-10T01:16:49.229 INFO:tasks.ceph.cluster2.osd.1.smithi043.stderr:2022-04-10T01:16:49.227+0000 7f181b6423c0 -1 osd.1 0 log_to_monitors true 2022-04-10T01:16:52.135 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2022-04-10T01:16:52.383 INFO:tasks.ceph.cluster2.osd.1.smithi043.stderr:2022-04-10T01:16:52.381+0000 7f181225d700 -1 osd.1 0 waiting for initial osdmap 2022-04-10T01:16:52.386 INFO:tasks.ceph.cluster2.osd.1.smithi043.stderr:2022-04-10T01:16:52.385+0000 7f180abef700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-04-10T01:16:52.401 INFO:tasks.ceph.cluster2.osd.2.smithi043.stderr:2022-04-10T01:16:52.399+0000 7f8bf14e2700 -1 osd.2 0 waiting for initial osdmap 2022-04-10T01:16:52.403 INFO:tasks.ceph.cluster2.osd.2.smithi043.stderr:2022-04-10T01:16:52.402+0000 7f8be9e74700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-04-10T01:16:52.415 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr:2022-04-10T01:16:52.414+0000 7f530d965700 -1 osd.0 0 waiting for initial osdmap 2022-04-10T01:16:52.419 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr:2022-04-10T01:16:52.417+0000 7f53062f7700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-04-10T01:16:52.522 INFO:teuthology.misc.health.smithi043.stdout: 2022-04-10T01:16:52.522 INFO:teuthology.misc.health.smithi043.stdout:{"epoch":7,"fsid":"1f0791d0-116c-427c-a4dc-de372595f89c","created":"2022-04-10T01:16:44.122952+0000","modified":"2022-04-10T01:16:51.097588+0000","last_up_change":"0.000000","last_in_change":"2022-04-10T01:16:45.085333+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"debf21d4-4a03-4599-9cb6-c88f44285145","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"28e8cb43-d9b1-4e2f-8639-138aa9881798","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"333de03e-cd2c-4d16-9535-027c7d0bac5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-04-10T01:16:52.535 DEBUG:teuthology.misc:0 of 3 OSDs are up 2022-04-10T01:16:58.838 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2022-04-10T01:16:59.151 INFO:teuthology.misc.health.smithi043.stdout: 2022-04-10T01:16:59.151 INFO:teuthology.misc.health.smithi043.stdout:{"epoch":12,"fsid":"1f0791d0-116c-427c-a4dc-de372595f89c","created":"2022-04-10T01:16:44.122952+0000","modified":"2022-04-10T01:16:58.070422+0000","last_up_change":"2022-04-10T01:16:53.398267+0000","last_in_change":"2022-04-10T01:16:45.085333+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-04-10T01:16:55.116409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"debf21d4-4a03-4599-9cb6-c88f44285145","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":34555},{"type":"v1","addr":"172.21.15.43:6801","nonce":34555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":34555},{"type":"v1","addr":"172.21.15.43:6803","nonce":34555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":34555},{"type":"v1","addr":"172.21.15.43:6807","nonce":34555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":34555},{"type":"v1","addr":"172.21.15.43:6805","nonce":34555}]},"public_addr":"172.21.15.43:6801/34555","cluster_addr":"172.21.15.43:6803/34555","heartbeat_back_addr":"172.21.15.43:6807/34555","heartbeat_front_addr":"172.21.15.43:6805/34555","state":["exists","up"]},{"osd":1,"uuid":"28e8cb43-d9b1-4e2f-8639-138aa9881798","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":34556},{"type":"v1","addr":"172.21.15.43:6809","nonce":34556}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":34556},{"type":"v1","addr":"172.21.15.43:6811","nonce":34556}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":34556},{"type":"v1","addr":"172.21.15.43:6819","nonce":34556}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":34556},{"type":"v1","addr":"172.21.15.43:6815","nonce":34556}]},"public_addr":"172.21.15.43:6809/34556","cluster_addr":"172.21.15.43:6811/34556","heartbeat_back_addr":"172.21.15.43:6819/34556","heartbeat_front_addr":"172.21.15.43:6815/34556","state":["exists","up"]},{"osd":2,"uuid":"333de03e-cd2c-4d16-9535-027c7d0bac5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":34557},{"type":"v1","addr":"172.21.15.43:6814","nonce":34557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":34557},{"type":"v1","addr":"172.21.15.43:6818","nonce":34557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":34557},{"type":"v1","addr":"172.21.15.43:6823","nonce":34557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":34557},{"type":"v1","addr":"172.21.15.43:6821","nonce":34557}]},"public_addr":"172.21.15.43:6814/34557","cluster_addr":"172.21.15.43:6818/34557","heartbeat_back_addr":"172.21.15.43:6823/34557","heartbeat_front_addr":"172.21.15.43:6821/34557","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-10T01:16:50.229424+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-10T01:16:50.214485+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-10T01:16:50.126252+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-04-10T01:16:59.163 DEBUG:teuthology.misc:3 of 3 OSDs are up 2022-04-10T01:16:59.164 INFO:tasks.ceph:Creating RBD pool 2022-04-10T01:16:59.164 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph --cluster cluster2 osd pool create rbd 8 2022-04-10T01:17:00.110 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'rbd' created 2022-04-10T01:17:00.123 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph --cluster cluster2 osd pool application enable rbd rbd --yes-i-really-mean-it 2022-04-10T01:17:01.114 INFO:teuthology.orchestra.run.smithi043.stderr:enabled application 'rbd' on pool 'rbd' 2022-04-10T01:17:01.127 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2022-04-10T01:17:01.128 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2022-04-10T01:17:01.128 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2022-04-10T01:17:01.129 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2022-04-10T01:17:01.129 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2022-04-10T01:17:01.496 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:17:01.508 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":5,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":39019},{"type":"v1","addr":"172.21.15.5:6827","nonce":39019}]},"active_addr":"172.21.15.5:6827/39019","active_change":"2022-04-10T01:16:51.051790+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.5:0","nonce":4223500457}]},{"addrvec":[{"type":"v2","addr":"172.21.15.5:0","nonce":2955533747}]},{"addrvec":[{"type":"v2","addr":"172.21.15.5:0","nonce":1880374769}]},{"addrvec":[{"type":"v2","addr":"172.21.15.5:0","nonce":3769334799}]}]}} 2022-04-10T01:17:01.512 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2022-04-10T01:17:01.512 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2022-04-10T01:17:01.513 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2022-04-10T01:17:01.821 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:17:01.821 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":14,"fsid":"1f0791d0-116c-427c-a4dc-de372595f89c","created":"2022-04-10T01:16:44.122952+0000","modified":"2022-04-10T01:17:01.100555+0000","last_up_change":"2022-04-10T01:16:53.398267+0000","last_in_change":"2022-04-10T01:16:45.085333+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-04-10T01:16:55.116409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-04-10T01:16:59.499465+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"debf21d4-4a03-4599-9cb6-c88f44285145","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":34555},{"type":"v1","addr":"172.21.15.43:6801","nonce":34555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":34555},{"type":"v1","addr":"172.21.15.43:6803","nonce":34555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":34555},{"type":"v1","addr":"172.21.15.43:6807","nonce":34555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":34555},{"type":"v1","addr":"172.21.15.43:6805","nonce":34555}]},"public_addr":"172.21.15.43:6801/34555","cluster_addr":"172.21.15.43:6803/34555","heartbeat_back_addr":"172.21.15.43:6807/34555","heartbeat_front_addr":"172.21.15.43:6805/34555","state":["exists","up"]},{"osd":1,"uuid":"28e8cb43-d9b1-4e2f-8639-138aa9881798","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":34556},{"type":"v1","addr":"172.21.15.43:6809","nonce":34556}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":34556},{"type":"v1","addr":"172.21.15.43:6811","nonce":34556}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":34556},{"type":"v1","addr":"172.21.15.43:6819","nonce":34556}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":34556},{"type":"v1","addr":"172.21.15.43:6815","nonce":34556}]},"public_addr":"172.21.15.43:6809/34556","cluster_addr":"172.21.15.43:6811/34556","heartbeat_back_addr":"172.21.15.43:6819/34556","heartbeat_front_addr":"172.21.15.43:6815/34556","state":["exists","up"]},{"osd":2,"uuid":"333de03e-cd2c-4d16-9535-027c7d0bac5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":34557},{"type":"v1","addr":"172.21.15.43:6814","nonce":34557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":34557},{"type":"v1","addr":"172.21.15.43:6818","nonce":34557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":34557},{"type":"v1","addr":"172.21.15.43:6823","nonce":34557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":34557},{"type":"v1","addr":"172.21.15.43:6821","nonce":34557}]},"public_addr":"172.21.15.43:6814/34557","cluster_addr":"172.21.15.43:6818/34557","heartbeat_back_addr":"172.21.15.43:6823/34557","heartbeat_front_addr":"172.21.15.43:6821/34557","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-10T01:16:50.229424+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-10T01:16:50.214485+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-10T01:16:50.126252+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-04-10T01:17:01.834 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2022-04-10T01:17:01.835 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2022-04-10T01:17:02.142 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:17:02.142 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":15,"fsid":"1f0791d0-116c-427c-a4dc-de372595f89c","created":"2022-04-10T01:16:44.122952+0000","modified":"2022-04-10T01:17:02.102677+0000","last_up_change":"2022-04-10T01:16:53.398267+0000","last_in_change":"2022-04-10T01:16:45.085333+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-04-10T01:16:55.116409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-04-10T01:16:59.499465+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"debf21d4-4a03-4599-9cb6-c88f44285145","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":34555},{"type":"v1","addr":"172.21.15.43:6801","nonce":34555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":34555},{"type":"v1","addr":"172.21.15.43:6803","nonce":34555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":34555},{"type":"v1","addr":"172.21.15.43:6807","nonce":34555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":34555},{"type":"v1","addr":"172.21.15.43:6805","nonce":34555}]},"public_addr":"172.21.15.43:6801/34555","cluster_addr":"172.21.15.43:6803/34555","heartbeat_back_addr":"172.21.15.43:6807/34555","heartbeat_front_addr":"172.21.15.43:6805/34555","state":["exists","up"]},{"osd":1,"uuid":"28e8cb43-d9b1-4e2f-8639-138aa9881798","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":34556},{"type":"v1","addr":"172.21.15.43:6809","nonce":34556}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":34556},{"type":"v1","addr":"172.21.15.43:6811","nonce":34556}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":34556},{"type":"v1","addr":"172.21.15.43:6819","nonce":34556}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":34556},{"type":"v1","addr":"172.21.15.43:6815","nonce":34556}]},"public_addr":"172.21.15.43:6809/34556","cluster_addr":"172.21.15.43:6811/34556","heartbeat_back_addr":"172.21.15.43:6819/34556","heartbeat_front_addr":"172.21.15.43:6815/34556","state":["exists","up"]},{"osd":2,"uuid":"333de03e-cd2c-4d16-9535-027c7d0bac5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":34557},{"type":"v1","addr":"172.21.15.43:6814","nonce":34557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":34557},{"type":"v1","addr":"172.21.15.43:6818","nonce":34557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":34557},{"type":"v1","addr":"172.21.15.43:6823","nonce":34557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":34557},{"type":"v1","addr":"172.21.15.43:6821","nonce":34557}]},"public_addr":"172.21.15.43:6814/34557","cluster_addr":"172.21.15.43:6818/34557","heartbeat_back_addr":"172.21.15.43:6823/34557","heartbeat_front_addr":"172.21.15.43:6821/34557","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-10T01:16:50.229424+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-10T01:16:50.214485+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-10T01:16:50.126252+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-04-10T01:17:02.156 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2022-04-10T01:17:02.157 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2022-04-10T01:17:02.157 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2022-04-10T01:17:02.324 INFO:teuthology.orchestra.run.smithi043.stdout:34359738372 2022-04-10T01:17:02.324 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2022-04-10T01:17:02.327 INFO:teuthology.orchestra.run.smithi043.stdout:34359738371 2022-04-10T01:17:02.328 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2022-04-10T01:17:02.329 INFO:teuthology.orchestra.run.smithi043.stdout:34359738373 2022-04-10T01:17:02.330 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2022-04-10T01:17:02.646 INFO:teuthology.orchestra.run.smithi043.stdout:34359738371 2022-04-10T01:17:02.661 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738372 got 34359738371 for osd.0 2022-04-10T01:17:02.688 INFO:teuthology.orchestra.run.smithi043.stdout:34359738372 2022-04-10T01:17:02.693 INFO:teuthology.orchestra.run.smithi043.stdout:34359738370 2022-04-10T01:17:02.704 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.2 2022-04-10T01:17:02.707 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738371 got 34359738370 for osd.1 2022-04-10T01:17:03.662 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2022-04-10T01:17:03.705 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2022-04-10T01:17:03.708 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2022-04-10T01:17:04.000 INFO:teuthology.orchestra.run.smithi043.stdout:34359738372 2022-04-10T01:17:04.016 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738372 got 34359738372 for osd.0 2022-04-10T01:17:04.016 DEBUG:teuthology.parallel:result is None 2022-04-10T01:17:04.044 INFO:teuthology.orchestra.run.smithi043.stdout:34359738373 2022-04-10T01:17:04.058 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738373 for osd.2 2022-04-10T01:17:04.059 DEBUG:teuthology.parallel:result is None 2022-04-10T01:17:04.084 INFO:teuthology.orchestra.run.smithi043.stdout:34359738371 2022-04-10T01:17:04.098 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738371 got 34359738371 for osd.1 2022-04-10T01:17:04.099 DEBUG:teuthology.parallel:result is None 2022-04-10T01:17:04.099 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2022-04-10T01:17:04.099 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2022-04-10T01:17:04.440 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:17:04.442 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2022-04-10T01:17:04.453 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2022-04-10T01:17:03.058298+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":37032,"kb_used_data":1296,"kb_used_omap":0,"kb_used_meta":35712,"kb_avail":283078488,"statfs":{"total":289910292480,"available":289872371712,"internally_reserved":0,"allocated":1327104,"data_stored":1041476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":36569088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.957714"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2022-04-10T01:17:01.111717+0000","last_change":"2022-04-10T01:17:01.109978+0000","last_active":"2022-04-10T01:17:01.111717+0000","last_peered":"2022-04-10T01:17:01.111717+0000","last_clean":"2022-04-10T01:17:01.111717+0000","last_became_active":"2022-04-10T01:17:01.109559+0000","last_became_peered":"2022-04-10T01:17:01.109559+0000","last_unstale":"2022-04-10T01:17:01.111717+0000","last_undegraded":"2022-04-10T01:17:01.111717+0000","last_fullsized":"2022-04-10T01:17:01.111717+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-10T01:17:00.097220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-10T01:17:00.097220+0000","last_clean_scrub_stamp":"2022-04-10T01:17:00.097220+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-11T01:49:50.571265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2022-04-10T01:17:01.113166+0000","last_change":"2022-04-10T01:17:01.111783+0000","last_active":"2022-04-10T01:17:01.113166+0000","last_peered":"2022-04-10T01:17:01.113166+0000","last_clean":"2022-04-10T01:17:01.113166+0000","last_became_active":"2022-04-10T01:17:01.111490+0000","last_became_peered":"2022-04-10T01:17:01.111490+0000","last_unstale":"2022-04-10T01:17:01.113166+0000","last_undegraded":"2022-04-10T01:17:01.113166+0000","last_fullsized":"2022-04-10T01:17:01.113166+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-10T01:17:00.097220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-10T01:17:00.097220+0000","last_clean_scrub_stamp":"2022-04-10T01:17:00.097220+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-11T02:54:29.928539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2022-04-10T01:17:01.111961+0000","last_change":"2022-04-10T01:17:01.111079+0000","last_active":"2022-04-10T01:17:01.111961+0000","last_peered":"2022-04-10T01:17:01.111961+0000","last_clean":"2022-04-10T01:17:01.111961+0000","last_became_active":"2022-04-10T01:17:01.110396+0000","last_became_peered":"2022-04-10T01:17:01.110396+0000","last_unstale":"2022-04-10T01:17:01.111961+0000","last_undegraded":"2022-04-10T01:17:01.111961+0000","last_fullsized":"2022-04-10T01:17:01.111961+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-10T01:17:00.097220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-10T01:17:00.097220+0000","last_clean_scrub_stamp":"2022-04-10T01:17:00.097220+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-11T01:36:19.921648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2022-04-10T01:17:01.112217+0000","last_change":"2022-04-10T01:17:01.112217+0000","last_active":"2022-04-10T01:17:01.112217+0000","last_peered":"2022-04-10T01:17:01.112217+0000","last_clean":"2022-04-10T01:17:01.112217+0000","last_became_active":"2022-04-10T01:17:01.111845+0000","last_became_peered":"2022-04-10T01:17:01.111845+0000","last_unstale":"2022-04-10T01:17:01.112217+0000","last_undegraded":"2022-04-10T01:17:01.112217+0000","last_fullsized":"2022-04-10T01:17:01.112217+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-10T01:17:00.097220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-10T01:17:00.097220+0000","last_clean_scrub_stamp":"2022-04-10T01:17:00.097220+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-11T11:55:46.902508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":139,"reported_epoch":14,"state":"active+clean","last_fresh":"2022-04-10T01:17:01.106929+0000","last_change":"2022-04-10T01:16:57.076166+0000","last_active":"2022-04-10T01:17:01.106929+0000","last_peered":"2022-04-10T01:17:01.106929+0000","last_clean":"2022-04-10T01:17:01.106929+0000","last_became_active":"2022-04-10T01:16:57.075000+0000","last_became_peered":"2022-04-10T01:16:57.075000+0000","last_unstale":"2022-04-10T01:17:01.106929+0000","last_undegraded":"2022-04-10T01:17:01.106929+0000","last_fullsized":"2022-04-10T01:17:01.106929+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-10T01:16:56.065587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-10T01:16:56.065587+0000","last_clean_scrub_stamp":"2022-04-10T01:16:56.065587+0000","objects_scrubbed":0,"log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-11T10:59:08.595278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2022-04-10T01:17:01.111448+0000","last_change":"2022-04-10T01:17:01.109652+0000","last_active":"2022-04-10T01:17:01.111448+0000","last_peered":"2022-04-10T01:17:01.111448+0000","last_clean":"2022-04-10T01:17:01.111448+0000","last_became_active":"2022-04-10T01:17:01.109346+0000","last_became_peered":"2022-04-10T01:17:01.109346+0000","last_unstale":"2022-04-10T01:17:01.111448+0000","last_undegraded":"2022-04-10T01:17:01.111448+0000","last_fullsized":"2022-04-10T01:17:01.111448+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-10T01:17:00.097220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-10T01:17:00.097220+0000","last_clean_scrub_stamp":"2022-04-10T01:17:00.097220+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-11T02:01:18.967833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2022-04-10T01:17:01.111130+0000","last_change":"2022-04-10T01:17:01.111130+0000","last_active":"2022-04-10T01:17:01.111130+0000","last_peered":"2022-04-10T01:17:01.111130+0000","last_clean":"2022-04-10T01:17:01.111130+0000","last_became_active":"2022-04-10T01:17:01.110751+0000","last_became_peered":"2022-04-10T01:17:01.110751+0000","last_unstale":"2022-04-10T01:17:01.111130+0000","last_undegraded":"2022-04-10T01:17:01.111130+0000","last_fullsized":"2022-04-10T01:17:01.111130+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-10T01:17:00.097220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-10T01:17:00.097220+0000","last_clean_scrub_stamp":"2022-04-10T01:17:00.097220+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-11T03:17:28.315290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2022-04-10T01:17:01.110989+0000","last_change":"2022-04-10T01:17:01.109973+0000","last_active":"2022-04-10T01:17:01.110989+0000","last_peered":"2022-04-10T01:17:01.110989+0000","last_clean":"2022-04-10T01:17:01.110989+0000","last_became_active":"2022-04-10T01:17:01.109668+0000","last_became_peered":"2022-04-10T01:17:01.109668+0000","last_unstale":"2022-04-10T01:17:01.110989+0000","last_undegraded":"2022-04-10T01:17:01.110989+0000","last_fullsized":"2022-04-10T01:17:01.110989+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-10T01:17:00.097220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-10T01:17:00.097220+0000","last_clean_scrub_stamp":"2022-04-10T01:17:00.097220+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-11T08:56:58.836351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2022-04-10T01:17:01.113199+0000","last_change":"2022-04-10T01:17:01.111949+0000","last_active":"2022-04-10T01:17:01.113199+0000","last_peered":"2022-04-10T01:17:01.113199+0000","last_clean":"2022-04-10T01:17:01.113199+0000","last_became_active":"2022-04-10T01:17:01.110401+0000","last_became_peered":"2022-04-10T01:17:01.110401+0000","last_unstale":"2022-04-10T01:17:01.113199+0000","last_undegraded":"2022-04-10T01:17:01.113199+0000","last_fullsized":"2022-04-10T01:17:01.113199+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-10T01:17:00.097220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-10T01:17:00.097220+0000","last_clean_scrub_stamp":"2022-04-10T01:17:00.097220+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-11T12:23:57.921902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":933912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11776,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":11648,"kb_avail":94360064,"statfs":{"total":96636764160,"available":96624705536,"internally_reserved":0,"allocated":122880,"data_stored":33544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":11927552},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27100000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12628,"kb_used_data":588,"kb_used_omap":0,"kb_used_meta":12032,"kb_avail":94359212,"statfs":{"total":96636764160,"available":96623833088,"internally_reserved":0,"allocated":602112,"data_stored":503966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":12320768},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28199999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26700000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12628,"kb_used_data":588,"kb_used_omap":0,"kb_used_meta":12032,"kb_avail":94359212,"statfs":{"total":96636764160,"available":96623833088,"internally_reserved":0,"allocated":602112,"data_stored":503966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":12320768},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-04-10T01:17:04.454 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2022-04-10T01:17:04.761 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:17:04.763 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2022-04-10T01:17:04.775 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2022-04-10T01:17:03.058298+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":37032,"kb_used_data":1296,"kb_used_omap":0,"kb_used_meta":35712,"kb_avail":283078488,"statfs":{"total":289910292480,"available":289872371712,"internally_reserved":0,"allocated":1327104,"data_stored":1041476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":36569088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.957714"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2022-04-10T01:17:01.111717+0000","last_change":"2022-04-10T01:17:01.109978+0000","last_active":"2022-04-10T01:17:01.111717+0000","last_peered":"2022-04-10T01:17:01.111717+0000","last_clean":"2022-04-10T01:17:01.111717+0000","last_became_active":"2022-04-10T01:17:01.109559+0000","last_became_peered":"2022-04-10T01:17:01.109559+0000","last_unstale":"2022-04-10T01:17:01.111717+0000","last_undegraded":"2022-04-10T01:17:01.111717+0000","last_fullsized":"2022-04-10T01:17:01.111717+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-10T01:17:00.097220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-10T01:17:00.097220+0000","last_clean_scrub_stamp":"2022-04-10T01:17:00.097220+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-11T01:49:50.571265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2022-04-10T01:17:01.113166+0000","last_change":"2022-04-10T01:17:01.111783+0000","last_active":"2022-04-10T01:17:01.113166+0000","last_peered":"2022-04-10T01:17:01.113166+0000","last_clean":"2022-04-10T01:17:01.113166+0000","last_became_active":"2022-04-10T01:17:01.111490+0000","last_became_peered":"2022-04-10T01:17:01.111490+0000","last_unstale":"2022-04-10T01:17:01.113166+0000","last_undegraded":"2022-04-10T01:17:01.113166+0000","last_fullsized":"2022-04-10T01:17:01.113166+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-10T01:17:00.097220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-10T01:17:00.097220+0000","last_clean_scrub_stamp":"2022-04-10T01:17:00.097220+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-11T02:54:29.928539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2022-04-10T01:17:01.111961+0000","last_change":"2022-04-10T01:17:01.111079+0000","last_active":"2022-04-10T01:17:01.111961+0000","last_peered":"2022-04-10T01:17:01.111961+0000","last_clean":"2022-04-10T01:17:01.111961+0000","last_became_active":"2022-04-10T01:17:01.110396+0000","last_became_peered":"2022-04-10T01:17:01.110396+0000","last_unstale":"2022-04-10T01:17:01.111961+0000","last_undegraded":"2022-04-10T01:17:01.111961+0000","last_fullsized":"2022-04-10T01:17:01.111961+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-10T01:17:00.097220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-10T01:17:00.097220+0000","last_clean_scrub_stamp":"2022-04-10T01:17:00.097220+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-11T01:36:19.921648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2022-04-10T01:17:01.112217+0000","last_change":"2022-04-10T01:17:01.112217+0000","last_active":"2022-04-10T01:17:01.112217+0000","last_peered":"2022-04-10T01:17:01.112217+0000","last_clean":"2022-04-10T01:17:01.112217+0000","last_became_active":"2022-04-10T01:17:01.111845+0000","last_became_peered":"2022-04-10T01:17:01.111845+0000","last_unstale":"2022-04-10T01:17:01.112217+0000","last_undegraded":"2022-04-10T01:17:01.112217+0000","last_fullsized":"2022-04-10T01:17:01.112217+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-10T01:17:00.097220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-10T01:17:00.097220+0000","last_clean_scrub_stamp":"2022-04-10T01:17:00.097220+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-11T11:55:46.902508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":139,"reported_epoch":14,"state":"active+clean","last_fresh":"2022-04-10T01:17:01.106929+0000","last_change":"2022-04-10T01:16:57.076166+0000","last_active":"2022-04-10T01:17:01.106929+0000","last_peered":"2022-04-10T01:17:01.106929+0000","last_clean":"2022-04-10T01:17:01.106929+0000","last_became_active":"2022-04-10T01:16:57.075000+0000","last_became_peered":"2022-04-10T01:16:57.075000+0000","last_unstale":"2022-04-10T01:17:01.106929+0000","last_undegraded":"2022-04-10T01:17:01.106929+0000","last_fullsized":"2022-04-10T01:17:01.106929+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-10T01:16:56.065587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-10T01:16:56.065587+0000","last_clean_scrub_stamp":"2022-04-10T01:16:56.065587+0000","objects_scrubbed":0,"log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-11T10:59:08.595278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2022-04-10T01:17:01.111448+0000","last_change":"2022-04-10T01:17:01.109652+0000","last_active":"2022-04-10T01:17:01.111448+0000","last_peered":"2022-04-10T01:17:01.111448+0000","last_clean":"2022-04-10T01:17:01.111448+0000","last_became_active":"2022-04-10T01:17:01.109346+0000","last_became_peered":"2022-04-10T01:17:01.109346+0000","last_unstale":"2022-04-10T01:17:01.111448+0000","last_undegraded":"2022-04-10T01:17:01.111448+0000","last_fullsized":"2022-04-10T01:17:01.111448+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-10T01:17:00.097220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-10T01:17:00.097220+0000","last_clean_scrub_stamp":"2022-04-10T01:17:00.097220+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-11T02:01:18.967833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2022-04-10T01:17:01.111130+0000","last_change":"2022-04-10T01:17:01.111130+0000","last_active":"2022-04-10T01:17:01.111130+0000","last_peered":"2022-04-10T01:17:01.111130+0000","last_clean":"2022-04-10T01:17:01.111130+0000","last_became_active":"2022-04-10T01:17:01.110751+0000","last_became_peered":"2022-04-10T01:17:01.110751+0000","last_unstale":"2022-04-10T01:17:01.111130+0000","last_undegraded":"2022-04-10T01:17:01.111130+0000","last_fullsized":"2022-04-10T01:17:01.111130+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-10T01:17:00.097220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-10T01:17:00.097220+0000","last_clean_scrub_stamp":"2022-04-10T01:17:00.097220+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-11T03:17:28.315290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2022-04-10T01:17:01.110989+0000","last_change":"2022-04-10T01:17:01.109973+0000","last_active":"2022-04-10T01:17:01.110989+0000","last_peered":"2022-04-10T01:17:01.110989+0000","last_clean":"2022-04-10T01:17:01.110989+0000","last_became_active":"2022-04-10T01:17:01.109668+0000","last_became_peered":"2022-04-10T01:17:01.109668+0000","last_unstale":"2022-04-10T01:17:01.110989+0000","last_undegraded":"2022-04-10T01:17:01.110989+0000","last_fullsized":"2022-04-10T01:17:01.110989+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-10T01:17:00.097220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-10T01:17:00.097220+0000","last_clean_scrub_stamp":"2022-04-10T01:17:00.097220+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-11T08:56:58.836351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2022-04-10T01:17:01.113199+0000","last_change":"2022-04-10T01:17:01.111949+0000","last_active":"2022-04-10T01:17:01.113199+0000","last_peered":"2022-04-10T01:17:01.113199+0000","last_clean":"2022-04-10T01:17:01.113199+0000","last_became_active":"2022-04-10T01:17:01.110401+0000","last_became_peered":"2022-04-10T01:17:01.110401+0000","last_unstale":"2022-04-10T01:17:01.113199+0000","last_undegraded":"2022-04-10T01:17:01.113199+0000","last_fullsized":"2022-04-10T01:17:01.113199+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-10T01:17:00.097220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-10T01:17:00.097220+0000","last_clean_scrub_stamp":"2022-04-10T01:17:00.097220+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-11T12:23:57.921902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":933912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11776,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":11648,"kb_avail":94360064,"statfs":{"total":96636764160,"available":96624705536,"internally_reserved":0,"allocated":122880,"data_stored":33544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":11927552},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27100000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12628,"kb_used_data":588,"kb_used_omap":0,"kb_used_meta":12032,"kb_avail":94359212,"statfs":{"total":96636764160,"available":96623833088,"internally_reserved":0,"allocated":602112,"data_stored":503966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":12320768},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28199999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26700000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12628,"kb_used_data":588,"kb_used_omap":0,"kb_used_meta":12032,"kb_avail":94359212,"statfs":{"total":96636764160,"available":96623833088,"internally_reserved":0,"allocated":602112,"data_stored":503966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":12320768},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":466956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-04-10T01:17:04.776 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2022-04-10T01:17:04.776 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2022-04-10T01:17:04.776 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2022-04-10T01:17:04.777 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2022-04-10T01:17:05.127 INFO:teuthology.orchestra.run.smithi043.stdout: 2022-04-10T01:17:05.127 INFO:teuthology.orchestra.run.smithi043.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-04-10T01:17:05.141 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2022-04-10T01:17:05.141 INFO:teuthology.run_tasks:Running task exec... 2022-04-10T01:17:05.154 INFO:teuthology.task.exec:Executing custom commands... 2022-04-10T01:17:05.154 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi043.front.sepia.ceph.com 2022-04-10T01:17:05.155 DEBUG:teuthology.orchestra.run.smithi043:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2022-04-10T01:17:05.546 INFO:teuthology.orchestra.run.smithi043.stderr:updated caps for client.mirror 2022-04-10T01:17:05.558 DEBUG:teuthology.orchestra.run.smithi043:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2022-04-10T01:17:05.953 INFO:teuthology.orchestra.run.smithi043.stderr:updated caps for client.mirror.0 2022-04-10T01:17:05.964 DEBUG:teuthology.orchestra.run.smithi043:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2022-04-10T01:17:06.355 INFO:teuthology.orchestra.run.smithi043.stderr:updated caps for client.mirror.1 2022-04-10T01:17:06.367 DEBUG:teuthology.orchestra.run.smithi043:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2022-04-10T01:17:06.761 INFO:teuthology.orchestra.run.smithi043.stderr:updated caps for client.mirror.2 2022-04-10T01:17:06.773 DEBUG:teuthology.orchestra.run.smithi043:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2022-04-10T01:17:07.166 INFO:teuthology.orchestra.run.smithi043.stderr:updated caps for client.mirror.3 2022-04-10T01:17:07.178 DEBUG:teuthology.orchestra.run.smithi043:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2022-04-10T01:17:07.569 INFO:teuthology.orchestra.run.smithi043.stderr:updated caps for client.mirror 2022-04-10T01:17:07.581 DEBUG:teuthology.orchestra.run.smithi043:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2022-04-10T01:17:07.973 INFO:teuthology.orchestra.run.smithi043.stderr:updated caps for client.mirror.0 2022-04-10T01:17:07.985 DEBUG:teuthology.orchestra.run.smithi043:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2022-04-10T01:17:08.382 INFO:teuthology.orchestra.run.smithi043.stderr:updated caps for client.mirror.1 2022-04-10T01:17:08.394 DEBUG:teuthology.orchestra.run.smithi043:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2022-04-10T01:17:08.787 INFO:teuthology.orchestra.run.smithi043.stderr:updated caps for client.mirror.2 2022-04-10T01:17:08.799 DEBUG:teuthology.orchestra.run.smithi043:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2022-04-10T01:17:09.198 INFO:teuthology.orchestra.run.smithi043.stderr:updated caps for client.mirror.3 2022-04-10T01:17:09.210 INFO:teuthology.run_tasks:Running task rbd-mirror... 2022-04-10T01:17:09.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2022-04-10T01:17:09.223 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2022-04-10T01:17:09.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2022-04-10T01:17:09.226 INFO:teuthology.run_tasks:Running task rbd-mirror... 2022-04-10T01:17:09.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2022-04-10T01:17:09.266 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2022-04-10T01:17:09.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2022-04-10T01:17:09.269 INFO:teuthology.run_tasks:Running task rbd-mirror... 2022-04-10T01:17:09.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2022-04-10T01:17:09.314 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2022-04-10T01:17:09.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2022-04-10T01:17:09.317 INFO:teuthology.run_tasks:Running task rbd-mirror... 2022-04-10T01:17:09.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2022-04-10T01:17:09.370 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2022-04-10T01:17:09.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2022-04-10T01:17:09.374 INFO:teuthology.run_tasks:Running task rbd-mirror... 2022-04-10T01:17:09.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2022-04-10T01:17:09.418 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2022-04-10T01:17:09.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2022-04-10T01:17:09.420 INFO:teuthology.run_tasks:Running task rbd-mirror... 2022-04-10T01:17:09.469 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2022-04-10T01:17:09.469 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2022-04-10T01:17:09.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2022-04-10T01:17:09.472 INFO:teuthology.run_tasks:Running task rbd-mirror... 2022-04-10T01:17:09.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2022-04-10T01:17:09.519 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2022-04-10T01:17:09.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2022-04-10T01:17:09.522 INFO:teuthology.run_tasks:Running task rbd-mirror... 2022-04-10T01:17:09.600 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2022-04-10T01:17:09.600 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2022-04-10T01:17:09.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2022-04-10T01:17:09.603 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2022-04-10T01:17:09.652 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1649553429 2022-04-10T01:17:09.652 DEBUG:tasks.rbd_mirror_thrash:Yielding 2022-04-10T01:17:09.652 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2022-04-10T01:17:09.700 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1649553429 2022-04-10T01:17:09.700 DEBUG:tasks.rbd_mirror_thrash:Yielding 2022-04-10T01:17:09.701 INFO:teuthology.run_tasks:Running task workunit... 2022-04-10T01:17:09.753 INFO:tasks.workunit:Pulling workunits from ref 9b3739e792ef182b79445b926c7b7da461c05f52 2022-04-10T01:17:09.755 INFO:tasks.workunit:Making a separate scratch dir for every client... 2022-04-10T01:17:09.756 DEBUG:teuthology.orchestra.run.smithi043:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2022-04-10T01:17:09.756 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster1 2022-04-10T01:17:09.757 INFO:tasks.rbd_mirror_thrash:waiting for 78 secs before thrashing 2022-04-10T01:17:09.757 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster2 2022-04-10T01:17:09.757 INFO:tasks.rbd_mirror_thrash:waiting for 84 secs before thrashing 2022-04-10T01:17:09.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-04-10T01:17:09.775 INFO:teuthology.orchestra.run.smithi043.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2022-04-10T01:17:09.775 DEBUG:teuthology.orchestra.run.smithi043:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2022-04-10T01:17:09.832 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2022-04-10T01:17:09.833 DEBUG:teuthology.orchestra.run.smithi043:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2022-04-10T01:17:09.889 INFO:tasks.workunit:timeout=6h 2022-04-10T01:17:09.889 INFO:tasks.workunit:cleanup=True 2022-04-10T01:17:09.890 DEBUG:teuthology.orchestra.run.smithi043:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone git://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout 9b3739e792ef182b79445b926c7b7da461c05f52 2022-04-10T01:17:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2022-04-10T01:17:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Note: switching to '9b3739e792ef182b79445b926c7b7da461c05f52'. 2022-04-10T01:17:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:17:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2022-04-10T01:17:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:changes and commit them, and you can discard any commits you make in this 2022-04-10T01:17:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:state without impacting any branches by switching back to a branch. 2022-04-10T01:17:41.764 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:17:41.764 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:If you want to create a new branch to retain commits you create, you may 2022-04-10T01:17:41.764 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:do so (now or later) by using -c with the switch command. Example: 2022-04-10T01:17:41.764 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:17:41.764 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: git switch -c 2022-04-10T01:17:41.765 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:17:41.765 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Or undo this operation with: 2022-04-10T01:17:41.765 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:17:41.765 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: git switch - 2022-04-10T01:17:41.766 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:17:41.766 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2022-04-10T01:17:41.766 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:17:41.766 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:HEAD is now at 9b3739e792e rbd-mirror: handle some inconsistency instead of asserting 2022-04-10T01:17:41.768 DEBUG:teuthology.orchestra.run.smithi043:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2022-04-10T01:17:41.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2022-04-10T01:17:41.837 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2022-04-10T01:17:41.837 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2022-04-10T01:17:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2022-04-10T01:17:42.473 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2022-04-10T01:17:42.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2022-04-10T01:17:42.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2022-04-10T01:17:42.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2022-04-10T01:17:42.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2022-04-10T01:17:42.556 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2022-04-10T01:17:42.556 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2022-04-10T01:17:42.614 INFO:tasks.workunit:Running workunits matching rbd/rbd_mirror_stress.sh on cluster1.client.mirror... 2022-04-10T01:17:42.616 INFO:tasks.workunit:Running workunit rbd/rbd_mirror_stress.sh... 2022-04-10T01:17:42.616 DEBUG:teuthology.orchestra.run.smithi043:workunit test rbd/rbd_mirror_stress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9b3739e792ef182b79445b926c7b7da461c05f52 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS='' MIRROR_IMAGE_MODE=snapshot MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering,exclusive-lock RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2022-04-10T01:17:42.681 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ IMAGE_COUNT=50 2022-04-10T01:17:42.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ export LOCKDEP=0 2022-04-10T01:17:42.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ LOCKDEP=0 2022-04-10T01:17:42.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2022-04-10T01:17:42.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2022-04-10T01:17:42.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ type xmlstarlet 2022-04-10T01:17:42.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ XMLSTARLET=xmlstarlet 2022-04-10T01:17:42.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ RBD_MIRROR_INSTANCES=4 2022-04-10T01:17:42.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ CLUSTER1=cluster1 2022-04-10T01:17:42.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ CLUSTER2=cluster2 2022-04-10T01:17:42.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ CLUSTER3=cluster3 2022-04-10T01:17:42.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ USE_CLUSTER3=0 2022-04-10T01:17:42.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ PEER_CLUSTER_SUFFIX= 2022-04-10T01:17:42.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ POOL=mirror 2022-04-10T01:17:42.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ PARENT_POOL=mirror_parent 2022-04-10T01:17:42.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ NS1=ns1 2022-04-10T01:17:42.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ NS2=ns2 2022-04-10T01:17:42.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ TEMPDIR= 2022-04-10T01:17:42.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ CEPH_ID=mirror 2022-04-10T01:17:42.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ RBD_IMAGE_FEATURES=layering,exclusive-lock 2022-04-10T01:17:42.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ MIRROR_USER_ID_PREFIX=mirror. 2022-04-10T01:17:42.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ MIRROR_POOL_MODE=image 2022-04-10T01:17:42.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ MIRROR_IMAGE_MODE=snapshot 2022-04-10T01:17:42.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ CLUSTERS='cluster1 cluster2' 2022-04-10T01:17:42.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 -eq 1 ']' 2022-04-10T01:17:42.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ export 'CEPH_ARGS=--id mirror' 2022-04-10T01:17:42.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ CEPH_ARGS='--id mirror' 2022-04-10T01:17:42.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ LAST_MIRROR_INSTANCE=3 2022-04-10T01:17:42.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2022-04-10T01:17:42.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2022-04-10T01:17:42.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2022-04-10T01:17:42.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ CEPH_BIN=. 2022-04-10T01:17:42.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ CEPH_SRC=. 2022-04-10T01:17:42.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' -e CMakeCache.txt ']' 2022-04-10T01:17:42.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 -gt 0 ']' 2022-04-10T01:17:42.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ setup 2022-04-10T01:17:42.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local c 2022-04-10T01:17:42.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ trap 'cleanup $?' INT TERM EXIT 2022-04-10T01:17:42.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ setup_tempdir 2022-04-10T01:17:42.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:17:42.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mktemp -d 2022-04-10T01:17:42.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ TEMPDIR=/tmp/tmp.rgwz74WspX 2022-04-10T01:17:42.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -z 1 ']' 2022-04-10T01:17:42.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ setup_pools cluster1 cluster2 cluster3 2022-04-10T01:17:42.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:17:42.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:17:42.698 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local 'remote_clusters=cluster2 cluster3' 2022-04-10T01:17:42.698 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mon_map_file 2022-04-10T01:17:42.698 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mon_addr 2022-04-10T01:17:42.698 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local admin_key_file 2022-04-10T01:17:42.699 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local uuid 2022-04-10T01:17:42.699 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:17:42.699 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:17:42.699 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:17:42.699 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:17:42.700 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ CEPH_ARGS= 2022-04-10T01:17:42.700 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ ceph --cluster cluster1 osd pool create mirror 64 64 2022-04-10T01:17:43.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:pool 'mirror' already exists 2022-04-10T01:17:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ CEPH_ARGS= 2022-04-10T01:17:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ ceph --cluster cluster1 osd pool create mirror_parent 64 64 2022-04-10T01:17:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:pool 'mirror_parent' already exists 2022-04-10T01:17:44.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ CEPH_ARGS= 2022-04-10T01:17:44.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 pool init mirror 2022-04-10T01:17:47.541 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ CEPH_ARGS= 2022-04-10T01:17:47.541 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 pool init mirror_parent 2022-04-10T01:17:50.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:17:50.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ CEPH_ARGS= 2022-04-10T01:17:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror image 2022-04-10T01:17:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2022-04-10T01:17:50.631 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2022-04-10T01:17:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2022-04-10T01:17:50.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 image 2022-04-10T01:17:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2022-04-10T01:17:50.769 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -z ']' 2022-04-10T01:17:50.769 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for remote_cluster in $remote_clusters 2022-04-10T01:17:50.769 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:17:50.770 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:17:50.770 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:17:50.770 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:17:50.770 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -z ']' 2022-04-10T01:17:50.771 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ peer_add cluster1 mirror cluster2 2022-04-10T01:17:50.771 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:17:50.771 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:17:50.771 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:17:50.771 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:17:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local client_cluster=cluster2 2022-04-10T01:17:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:17:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:17:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local uuid_var_name 2022-04-10T01:17:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:17:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local error_code 2022-04-10T01:17:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local peer_uuid 2022-04-10T01:17:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 16 32 2022-04-10T01:17:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:17:50.774 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2022-04-10T01:17:50.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ peer_uuid=08c3f2b1-864c-4af5-a27f-6f73cb9662e5 2022-04-10T01:17:50.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=0 2022-04-10T01:17:50.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:17:50.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 0 -eq 17 ']' 2022-04-10T01:17:50.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 0 -eq 0 2022-04-10T01:17:50.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:17:50.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:17:50.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ peer_add cluster1 mirror_parent cluster2 2022-04-10T01:17:50.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:17:50.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:17:50.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror_parent 2022-04-10T01:17:50.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:17:50.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local client_cluster=cluster2 2022-04-10T01:17:50.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:17:50.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:17:50.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local uuid_var_name 2022-04-10T01:17:50.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:17:50.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local error_code 2022-04-10T01:17:50.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local peer_uuid 2022-04-10T01:17:50.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 16 32 2022-04-10T01:17:50.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:17:50.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2022-04-10T01:17:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ peer_uuid=9954a0f0-f388-48d6-b8a7-3f70a3fc308c 2022-04-10T01:17:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=0 2022-04-10T01:17:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:17:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 0 -eq 17 ']' 2022-04-10T01:17:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 0 -eq 0 2022-04-10T01:17:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:17:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:17:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for remote_cluster in $remote_clusters 2022-04-10T01:17:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster3 2022-04-10T01:17:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster3 2022-04-10T01:17:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster3 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:17:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 1 2022-04-10T01:17:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ continue 2022-04-10T01:17:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ setup_pools cluster2 cluster1 cluster3 2022-04-10T01:17:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:17:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:17:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local 'remote_clusters=cluster1 cluster3' 2022-04-10T01:17:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mon_map_file 2022-04-10T01:17:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mon_addr 2022-04-10T01:17:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local admin_key_file 2022-04-10T01:17:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local uuid 2022-04-10T01:17:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:17:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:17:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:17:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:17:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ CEPH_ARGS= 2022-04-10T01:17:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ ceph --cluster cluster2 osd pool create mirror 64 64 2022-04-10T01:17:51.327 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:pool 'mirror' already exists 2022-04-10T01:17:51.336 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ CEPH_ARGS= 2022-04-10T01:17:51.337 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ ceph --cluster cluster2 osd pool create mirror_parent 64 64 2022-04-10T01:17:52.355 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:pool 'mirror_parent' already exists 2022-04-10T01:17:52.364 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ CEPH_ARGS= 2022-04-10T01:17:52.365 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 pool init mirror 2022-04-10T01:17:55.222 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ CEPH_ARGS= 2022-04-10T01:17:55.223 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 pool init mirror_parent 2022-04-10T01:17:58.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:17:58.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ CEPH_ARGS= 2022-04-10T01:17:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror image 2022-04-10T01:17:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2022-04-10T01:17:58.313 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2022-04-10T01:17:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2022-04-10T01:17:58.376 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 image 2022-04-10T01:17:58.409 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2022-04-10T01:17:58.445 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -z ']' 2022-04-10T01:17:58.445 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for remote_cluster in $remote_clusters 2022-04-10T01:17:58.446 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:17:58.446 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:17:58.446 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:17:58.446 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:17:58.447 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -z ']' 2022-04-10T01:17:58.447 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ peer_add cluster2 mirror cluster1 2022-04-10T01:17:58.447 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:17:58.447 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:17:58.448 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:17:58.448 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:17:58.448 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local client_cluster=cluster1 2022-04-10T01:17:58.448 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:17:58.448 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster1 2022-04-10T01:17:58.449 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local uuid_var_name 2022-04-10T01:17:58.449 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:17:58.449 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local error_code 2022-04-10T01:17:58.449 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local peer_uuid 2022-04-10T01:17:58.450 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 16 32 2022-04-10T01:17:58.450 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:17:58.450 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2022-04-10T01:17:58.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ peer_uuid=ac61d79b-3c20-443f-8f0d-3ce84b8d0ac8 2022-04-10T01:17:58.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=0 2022-04-10T01:17:58.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:17:58.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 0 -eq 17 ']' 2022-04-10T01:17:58.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 0 -eq 0 2022-04-10T01:17:58.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:17:58.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:17:58.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ peer_add cluster2 mirror_parent cluster1 2022-04-10T01:17:58.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:17:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:17:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror_parent 2022-04-10T01:17:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:17:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local client_cluster=cluster1 2022-04-10T01:17:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:17:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster1 2022-04-10T01:17:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local uuid_var_name 2022-04-10T01:17:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:17:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local error_code 2022-04-10T01:17:58.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local peer_uuid 2022-04-10T01:17:58.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 16 32 2022-04-10T01:17:58.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:17:58.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2022-04-10T01:17:58.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ peer_uuid=589782a3-9a22-4bc5-8fac-7ae7de5f6879 2022-04-10T01:17:58.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=0 2022-04-10T01:17:58.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:17:58.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 0 -eq 17 ']' 2022-04-10T01:17:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 0 -eq 0 2022-04-10T01:17:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:17:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:17:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for remote_cluster in $remote_clusters 2022-04-10T01:17:58.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster3 2022-04-10T01:17:58.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster3 2022-04-10T01:17:58.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster3 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:17:58.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 1 2022-04-10T01:17:58.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ continue 2022-04-10T01:17:58.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ setup_pools cluster3 cluster1 cluster2 2022-04-10T01:17:58.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster3 2022-04-10T01:17:58.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:17:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local 'remote_clusters=cluster1 cluster2' 2022-04-10T01:17:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mon_map_file 2022-04-10T01:17:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mon_addr 2022-04-10T01:17:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local admin_key_file 2022-04-10T01:17:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local uuid 2022-04-10T01:17:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster3 2022-04-10T01:17:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster3 2022-04-10T01:17:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster3 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:17:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 1 2022-04-10T01:17:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:17:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:17:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ start_mirrors cluster1 2022-04-10T01:17:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:17:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:17:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:17:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:17:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:17:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ seq 0 3 2022-04-10T01:17:58.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:17:58.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ start_mirror cluster1:0 2022-04-10T01:17:58.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1:0 2022-04-10T01:17:58.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local instance 2022-04-10T01:17:58.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:17:58.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local val=cluster1:0 2022-04-10T01:17:58.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster_var_name=cluster 2022-04-10T01:17:58.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local instance_var_name=instance 2022-04-10T01:17:58.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cluster=cluster1 2022-04-10T01:17:58.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ instance=0 2022-04-10T01:17:58.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 0 = cluster1:0 ']' 2022-04-10T01:17:58.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ eval cluster=cluster1 2022-04-10T01:17:58.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:17:58.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ eval instance=0 2022-04-10T01:17:58.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:17:58.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:17:58.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:17:58.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:17:58.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:17:58.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n 1 2022-04-10T01:17:58.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 2022-04-10T01:17:58.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:17:58.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ start_mirror cluster1:1 2022-04-10T01:17:58.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1:1 2022-04-10T01:17:58.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local instance 2022-04-10T01:17:58.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set_cluster_instance cluster1:1 cluster instance 2022-04-10T01:17:58.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local val=cluster1:1 2022-04-10T01:17:58.541 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster_var_name=cluster 2022-04-10T01:17:58.541 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local instance_var_name=instance 2022-04-10T01:17:58.541 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cluster=cluster1 2022-04-10T01:17:58.541 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ instance=1 2022-04-10T01:17:58.541 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 1 = cluster1:1 ']' 2022-04-10T01:17:58.542 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ eval cluster=cluster1 2022-04-10T01:17:58.542 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:17:58.542 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ eval instance=1 2022-04-10T01:17:58.542 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=1 2022-04-10T01:17:58.543 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:17:58.543 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:17:58.543 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:17:58.543 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:17:58.543 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n 1 2022-04-10T01:17:58.544 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 2022-04-10T01:17:58.544 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:17:58.544 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ start_mirror cluster1:2 2022-04-10T01:17:58.544 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1:2 2022-04-10T01:17:58.544 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local instance 2022-04-10T01:17:58.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set_cluster_instance cluster1:2 cluster instance 2022-04-10T01:17:58.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local val=cluster1:2 2022-04-10T01:17:58.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster_var_name=cluster 2022-04-10T01:17:58.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local instance_var_name=instance 2022-04-10T01:17:58.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cluster=cluster1 2022-04-10T01:17:58.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ instance=2 2022-04-10T01:17:58.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 2 = cluster1:2 ']' 2022-04-10T01:17:58.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ eval cluster=cluster1 2022-04-10T01:17:58.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:17:58.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ eval instance=2 2022-04-10T01:17:58.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=2 2022-04-10T01:17:58.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:17:58.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:17:58.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:17:58.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:17:58.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n 1 2022-04-10T01:17:58.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 2022-04-10T01:17:58.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:17:58.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ start_mirror cluster1:3 2022-04-10T01:17:58.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1:3 2022-04-10T01:17:58.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local instance 2022-04-10T01:17:58.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set_cluster_instance cluster1:3 cluster instance 2022-04-10T01:17:58.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local val=cluster1:3 2022-04-10T01:17:58.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster_var_name=cluster 2022-04-10T01:17:58.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local instance_var_name=instance 2022-04-10T01:17:58.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cluster=cluster1 2022-04-10T01:17:58.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ instance=3 2022-04-10T01:17:58.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 3 = cluster1:3 ']' 2022-04-10T01:17:58.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ eval cluster=cluster1 2022-04-10T01:17:58.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:17:58.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ eval instance=3 2022-04-10T01:17:58.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=3 2022-04-10T01:17:58.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:17:58.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:17:58.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:17:58.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:17:58.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n 1 2022-04-10T01:17:58.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 2022-04-10T01:17:58.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ start_mirrors cluster2 2022-04-10T01:17:58.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:17:58.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:17:58.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:17:58.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:17:58.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:17:58.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ seq 0 3 2022-04-10T01:17:58.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:17:58.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ start_mirror cluster2:0 2022-04-10T01:17:58.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2:0 2022-04-10T01:17:58.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local instance 2022-04-10T01:17:58.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set_cluster_instance cluster2:0 cluster instance 2022-04-10T01:17:58.556 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local val=cluster2:0 2022-04-10T01:17:58.556 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster_var_name=cluster 2022-04-10T01:17:58.556 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local instance_var_name=instance 2022-04-10T01:17:58.556 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cluster=cluster2 2022-04-10T01:17:58.556 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ instance=0 2022-04-10T01:17:58.557 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 0 = cluster2:0 ']' 2022-04-10T01:17:58.557 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ eval cluster=cluster2 2022-04-10T01:17:58.557 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster2 2022-04-10T01:17:58.557 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ eval instance=0 2022-04-10T01:17:58.557 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:17:58.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:17:58.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:17:58.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:17:58.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:17:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n 1 2022-04-10T01:17:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 2022-04-10T01:17:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:17:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ start_mirror cluster2:1 2022-04-10T01:17:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2:1 2022-04-10T01:17:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local instance 2022-04-10T01:17:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set_cluster_instance cluster2:1 cluster instance 2022-04-10T01:17:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local val=cluster2:1 2022-04-10T01:17:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster_var_name=cluster 2022-04-10T01:17:58.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local instance_var_name=instance 2022-04-10T01:17:58.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cluster=cluster2 2022-04-10T01:17:58.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ instance=1 2022-04-10T01:17:58.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 1 = cluster2:1 ']' 2022-04-10T01:17:58.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ eval cluster=cluster2 2022-04-10T01:17:58.562 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster2 2022-04-10T01:17:58.562 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ eval instance=1 2022-04-10T01:17:58.562 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=1 2022-04-10T01:17:58.562 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:17:58.562 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:17:58.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:17:58.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:17:58.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n 1 2022-04-10T01:17:58.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 2022-04-10T01:17:58.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:17:58.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ start_mirror cluster2:2 2022-04-10T01:17:58.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2:2 2022-04-10T01:17:58.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local instance 2022-04-10T01:17:58.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set_cluster_instance cluster2:2 cluster instance 2022-04-10T01:17:58.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local val=cluster2:2 2022-04-10T01:17:58.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster_var_name=cluster 2022-04-10T01:17:58.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local instance_var_name=instance 2022-04-10T01:17:58.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cluster=cluster2 2022-04-10T01:17:58.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ instance=2 2022-04-10T01:17:58.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 2 = cluster2:2 ']' 2022-04-10T01:17:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ eval cluster=cluster2 2022-04-10T01:17:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster2 2022-04-10T01:17:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ eval instance=2 2022-04-10T01:17:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=2 2022-04-10T01:17:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:17:58.567 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:17:58.567 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:17:58.567 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:17:58.567 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n 1 2022-04-10T01:17:58.567 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 2022-04-10T01:17:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:17:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ start_mirror cluster2:3 2022-04-10T01:17:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2:3 2022-04-10T01:17:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local instance 2022-04-10T01:17:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set_cluster_instance cluster2:3 cluster instance 2022-04-10T01:17:58.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local val=cluster2:3 2022-04-10T01:17:58.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster_var_name=cluster 2022-04-10T01:17:58.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local instance_var_name=instance 2022-04-10T01:17:58.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cluster=cluster2 2022-04-10T01:17:58.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ instance=3 2022-04-10T01:17:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 3 = cluster2:3 ']' 2022-04-10T01:17:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ eval cluster=cluster2 2022-04-10T01:17:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster2 2022-04-10T01:17:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ eval instance=3 2022-04-10T01:17:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=3 2022-04-10T01:17:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:17:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:17:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:17:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:17:58.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n 1 2022-04-10T01:17:58.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 2022-04-10T01:17:58.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ testlog 'TEST: add image and test replay after client crashes' 2022-04-10T01:17:58.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ tee -a /tmp/tmp.rgwz74WspX/rbd-mirror.test.log 2022-04-10T01:17:58.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ date '+%F %T' 2022-04-10T01:17:58.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo 2022-04-10 01:17:58 TEST: add image and test replay after client crashes 2022-04-10T01:17:58.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:2022-04-10 01:17:58 TEST: add image and test replay after client crashes 2022-04-10T01:17:58.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=test 2022-04-10T01:17:58.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image_and_enable_mirror cluster2 mirror test snapshot 512M 2022-04-10T01:17:58.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:17:58.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:17:58.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:17:58.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:17:58.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:17:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:17:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:17:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:17:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:17:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:17:58.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:17:58.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snapshot ']' 2022-04-10T01:17:58.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:17:58.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror test 512M 2022-04-10T01:17:58.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:17:58.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:17:58.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:17:58.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:17:58.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:17:58.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:17:58.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:17:58.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:17:58.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:17:58.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:17:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:17:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 512M ']' 2022-04-10T01:17:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=512M 2022-04-10T01:17:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:17:58.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/test 2022-04-10T01:17:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' image = image ']' 2022-04-10T01:17:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror test snapshot 2022-04-10T01:17:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:17:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:17:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:17:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:17:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:17:58.583 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:17:58.583 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:17:58.583 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:17:58.583 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test snapshot 2022-04-10T01:17:59.293 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:17:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/test 2022-04-10T01:17:59.337 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'test': 2022-04-10T01:17:59.337 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 512 MiB in 128 objects 2022-04-10T01:17:59.337 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:17:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:17:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 10a4ed5e1bfc 2022-04-10T01:17:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.10a4ed5e1bfc 2022-04-10T01:17:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:17:59.339 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:17:59.339 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:17:59.339 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:17:59.339 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:17:58 2022 2022-04-10T01:17:59.339 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:17:58 2022 2022-04-10T01:17:59.340 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:17:58 2022 2022-04-10T01:17:59.340 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:17:59.340 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:17:59.340 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 5ff848f4-a6a0-498d-b9f1-7a041de7de98 2022-04-10T01:17:59.341 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:17:59.342 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror test 2022-04-10T01:17:59.343 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:17:59.343 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:17:59.343 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:17:59.343 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:17:59.344 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:17:59.344 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:17:59.344 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:17:59.344 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2022-04-10T01:17:59.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:17:59.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:17:59.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:17:59.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:17:59.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:17:59.346 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:17:59.346 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:17:59.346 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:17:59.346 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:17:59.347 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:17:59.347 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:18:00.353 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror test started 2022-04-10T01:18:00.353 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:18:00.354 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:18:00.354 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:18:00.355 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:18:00.355 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:18:00.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:18:00.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:18:00.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:18:00.357 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:18:00.357 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:18:00.358 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2022-04-10T01:18:00.358 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:18:00.359 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:18:00.359 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:18:00.359 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:18:00.360 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:18:00.360 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:18:00.361 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:18:00.361 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:18:00.362 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:18:00.362 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:18:00.362 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:18:00.363 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:18:00.363 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:18:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:18:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:18:00.365 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2022-04-10T01:18:00.365 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:18:00.366 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:18:00.366 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:18:00.366 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:18:00.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:18:00.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:18:00.368 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:18:00.368 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:18:00.369 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:18:00.369 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:18:00.369 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:18:00.370 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:18:00.370 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:18:00.371 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:18:00.371 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:18:00.372 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:18:00.372 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:18:00.372 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:18:00.373 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:18:00.373 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:18:00.374 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:18:00.374 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:18:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:18:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:18:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:18:00.376 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:18:00.376 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:18:00.377 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:18:00.377 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:18:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:18:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:18:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:18:00.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:18:00.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:18:00.380 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:18:00.380 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:18:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:18:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:18:00.383 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:18:00.383 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:18:00.384 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:18:00.384 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2022-04-10T01:18:00.465 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/test'] 2022-04-10T01:18:00.465 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:18:00.466 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:18:00.474 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:18:00.474 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2022-04-10T01:18:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:1 2022-04-10T01:18:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:18:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:18:00.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:1 cluster instance 2022-04-10T01:18:00.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:1 2022-04-10T01:18:00.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:18:00.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:18:00.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:18:00.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=1 2022-04-10T01:18:00.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 1 = cluster1:1 ']' 2022-04-10T01:18:00.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:18:00.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:18:00.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=1 2022-04-10T01:18:00.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:18:00.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:18:00.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:18:00.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:18:00.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:18:00.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:1 cluster1 2022-04-10T01:18:00.483 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:1 2022-04-10T01:18:00.483 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:18:00.484 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:18:00.484 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2022-04-10T01:18:00.484 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:1 2022-04-10T01:18:00.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:18:00.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:18:00.486 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:18:00.486 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:18:00.487 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 1 = cluster1:1 ']' 2022-04-10T01:18:00.487 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:18:00.487 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:18:00.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=1 2022-04-10T01:18:00.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=1 2022-04-10T01:18:00.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:18:00.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:18:00.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:18:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:18:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2022-04-10T01:18:00.490 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:18:00.490 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:18:00.490 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:18:00.491 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2022-04-10T01:18:00.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/test'] 2022-04-10T01:18:00.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:18:00.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:18:00.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:18:00.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2022-04-10T01:18:00.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:2 2022-04-10T01:18:00.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:18:00.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:18:00.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:2 cluster instance 2022-04-10T01:18:00.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:2 2022-04-10T01:18:00.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:18:00.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:18:00.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:18:00.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=2 2022-04-10T01:18:00.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 2 = cluster1:2 ']' 2022-04-10T01:18:00.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:18:00.583 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:18:00.583 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=2 2022-04-10T01:18:00.583 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:18:00.583 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:18:00.583 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:18:00.584 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:18:00.584 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:18:00.584 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:2 cluster1 2022-04-10T01:18:00.584 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:2 2022-04-10T01:18:00.584 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:18:00.585 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:18:00.585 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2022-04-10T01:18:00.585 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:2 2022-04-10T01:18:00.585 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:18:00.586 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:18:00.586 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:18:00.586 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:18:00.586 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 2 = cluster1:2 ']' 2022-04-10T01:18:00.586 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:18:00.587 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:18:00.587 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=2 2022-04-10T01:18:00.587 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=2 2022-04-10T01:18:00.587 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:18:00.588 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:18:00.588 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:18:00.588 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:18:00.588 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2022-04-10T01:18:00.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:18:00.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:18:00.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:18:00.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2022-04-10T01:18:00.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/test'] 2022-04-10T01:18:00.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:18:00.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:18:00.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:18:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2022-04-10T01:18:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:3 2022-04-10T01:18:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:18:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:18:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:3 cluster instance 2022-04-10T01:18:00.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:3 2022-04-10T01:18:00.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:18:00.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:18:00.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:18:00.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=3 2022-04-10T01:18:00.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 3 = cluster1:3 ']' 2022-04-10T01:18:00.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:18:00.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:18:00.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=3 2022-04-10T01:18:00.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=3 2022-04-10T01:18:00.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:18:00.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:18:00.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:18:00.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:18:00.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:3 cluster1 2022-04-10T01:18:00.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:3 2022-04-10T01:18:00.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:18:00.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:18:00.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2022-04-10T01:18:00.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:3 2022-04-10T01:18:00.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:18:00.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:18:00.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:18:00.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=3 2022-04-10T01:18:00.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 3 = cluster1:3 ']' 2022-04-10T01:18:00.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:18:00.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:18:00.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=3 2022-04-10T01:18:00.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=3 2022-04-10T01:18:00.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:18:00.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:18:00.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:18:00.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:18:00.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2022-04-10T01:18:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:18:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:18:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:18:00.703 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2022-04-10T01:18:00.792 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/test'] 2022-04-10T01:18:00.793 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:18:00.793 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:18:00.794 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:18:00.794 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 1 2022-04-10T01:18:01.795 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:18:01.795 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:18:01.799 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:18:01.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2022-04-10T01:18:01.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:18:01.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:18:01.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:18:01.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:18:01.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:18:01.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:18:01.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:18:01.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:18:01.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:18:01.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:18:01.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:18:01.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:18:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:18:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:18:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:18:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:18:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:18:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:18:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:18:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:18:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:18:01.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:18:01.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:18:01.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:18:01.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:18:01.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:18:01.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:18:01.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:18:01.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:18:01.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:18:01.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:18:01.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:18:01.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:18:01.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:18:01.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:18:01.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:18:01.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:18:01.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:18:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:18:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:18:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:18:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2022-04-10T01:18:01.916 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/test'] 2022-04-10T01:18:01.916 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:18:01.916 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:18:01.920 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:18:01.921 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2022-04-10T01:18:01.921 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:1 2022-04-10T01:18:01.921 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:18:01.921 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:18:01.922 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:1 cluster instance 2022-04-10T01:18:01.922 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:1 2022-04-10T01:18:01.922 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:18:01.922 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:18:01.923 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:18:01.923 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=1 2022-04-10T01:18:01.923 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 1 = cluster1:1 ']' 2022-04-10T01:18:01.923 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:18:01.923 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:18:01.924 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=1 2022-04-10T01:18:01.924 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:18:01.924 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:18:01.924 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:18:01.925 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:18:01.925 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:18:01.925 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:1 cluster1 2022-04-10T01:18:01.925 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:1 2022-04-10T01:18:01.925 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:18:01.926 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:18:01.926 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2022-04-10T01:18:01.926 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:1 2022-04-10T01:18:01.926 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:18:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:18:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:18:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:18:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 1 = cluster1:1 ']' 2022-04-10T01:18:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:18:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:18:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=1 2022-04-10T01:18:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=1 2022-04-10T01:18:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:18:01.929 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:18:01.929 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:18:01.929 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:18:01.929 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2022-04-10T01:18:01.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:18:01.935 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:18:01.935 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:18:01.935 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2022-04-10T01:18:02.025 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/test'] 2022-04-10T01:18:02.026 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:18:02.026 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:18:02.030 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:18:02.030 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2022-04-10T01:18:02.030 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:2 2022-04-10T01:18:02.031 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:18:02.031 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:18:02.031 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:2 cluster instance 2022-04-10T01:18:02.031 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:2 2022-04-10T01:18:02.032 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:18:02.032 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:18:02.032 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:18:02.032 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=2 2022-04-10T01:18:02.032 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 2 = cluster1:2 ']' 2022-04-10T01:18:02.033 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:18:02.033 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:18:02.033 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=2 2022-04-10T01:18:02.033 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:18:02.033 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:18:02.034 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:18:02.034 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:18:02.034 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:18:02.034 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:2 cluster1 2022-04-10T01:18:02.035 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:2 2022-04-10T01:18:02.035 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:18:02.035 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:18:02.035 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2022-04-10T01:18:02.035 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:2 2022-04-10T01:18:02.036 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:18:02.036 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:18:02.036 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:18:02.036 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:18:02.036 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 2 = cluster1:2 ']' 2022-04-10T01:18:02.037 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:18:02.037 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:18:02.037 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=2 2022-04-10T01:18:02.037 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=2 2022-04-10T01:18:02.037 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:18:02.038 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:18:02.038 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:18:02.038 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:18:02.038 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2022-04-10T01:18:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:18:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:18:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:18:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2022-04-10T01:18:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/test'] 2022-04-10T01:18:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:18:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:18:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:18:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2022-04-10T01:18:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:3 2022-04-10T01:18:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:18:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:18:02.140 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:3 cluster instance 2022-04-10T01:18:02.140 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:3 2022-04-10T01:18:02.140 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:18:02.140 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:18:02.141 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:18:02.141 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=3 2022-04-10T01:18:02.141 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 3 = cluster1:3 ']' 2022-04-10T01:18:02.141 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:18:02.141 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:18:02.142 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=3 2022-04-10T01:18:02.142 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=3 2022-04-10T01:18:02.142 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:18:02.142 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:18:02.143 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:18:02.143 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:18:02.143 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:3 cluster1 2022-04-10T01:18:02.143 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:3 2022-04-10T01:18:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:18:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:18:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2022-04-10T01:18:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:3 2022-04-10T01:18:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:18:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:18:02.145 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:18:02.145 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=3 2022-04-10T01:18:02.145 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 3 = cluster1:3 ']' 2022-04-10T01:18:02.145 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:18:02.146 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:18:02.146 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=3 2022-04-10T01:18:02.146 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=3 2022-04-10T01:18:02.146 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:18:02.147 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:18:02.147 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:18:02.147 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:18:02.147 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2022-04-10T01:18:02.152 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:18:02.152 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:18:02.153 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:18:02.153 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2022-04-10T01:18:02.243 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/test'] 2022-04-10T01:18:02.243 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:18:02.243 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:18:02.244 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:18:02.244 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 2 2022-04-10T01:18:04.245 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:18:04.246 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:18:04.250 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:18:04.251 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2022-04-10T01:18:04.251 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:18:04.251 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:18:04.252 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:18:04.252 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:18:04.252 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:18:04.252 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:18:04.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:18:04.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:18:04.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:18:04.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:18:04.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:18:04.254 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:18:04.254 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:18:04.254 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:18:04.254 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:18:04.255 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:18:04.255 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:18:04.255 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:18:04.255 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:18:04.256 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:18:04.256 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:18:04.256 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:18:04.256 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:18:04.256 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:18:04.257 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:18:04.257 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:18:04.257 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:18:04.257 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:18:04.257 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:18:04.258 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:18:04.258 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:18:04.258 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:18:04.258 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:18:04.259 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:18:04.259 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:18:04.259 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:18:04.259 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:18:04.259 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:18:04.267 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:18:04.267 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:18:04.267 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:18:04.267 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2022-04-10T01:18:04.365 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/test'] 2022-04-10T01:18:04.366 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:18:04.366 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:18:04.370 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:18:04.370 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2022-04-10T01:18:04.371 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:1 2022-04-10T01:18:04.371 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:18:04.371 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:18:04.371 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:1 cluster instance 2022-04-10T01:18:04.372 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:1 2022-04-10T01:18:04.372 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:18:04.372 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:18:04.372 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:18:04.373 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=1 2022-04-10T01:18:04.373 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 1 = cluster1:1 ']' 2022-04-10T01:18:04.373 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:18:04.373 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:18:04.374 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=1 2022-04-10T01:18:04.374 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:18:04.374 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:18:04.374 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:18:04.375 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:18:04.375 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:18:04.375 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:1 cluster1 2022-04-10T01:18:04.375 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:1 2022-04-10T01:18:04.375 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:18:04.376 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:18:04.376 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2022-04-10T01:18:04.376 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:1 2022-04-10T01:18:04.376 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:18:04.377 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:18:04.377 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:18:04.377 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:18:04.377 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 1 = cluster1:1 ']' 2022-04-10T01:18:04.377 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:18:04.378 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:18:04.378 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=1 2022-04-10T01:18:04.378 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=1 2022-04-10T01:18:04.378 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:18:04.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:18:04.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:18:04.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:18:04.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2022-04-10T01:18:04.385 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:18:04.385 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:18:04.386 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:18:04.386 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2022-04-10T01:18:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/test'] 2022-04-10T01:18:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:18:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:18:04.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:18:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2022-04-10T01:18:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:2 2022-04-10T01:18:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:18:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:18:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:2 cluster instance 2022-04-10T01:18:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:2 2022-04-10T01:18:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:18:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:18:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:18:04.483 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=2 2022-04-10T01:18:04.483 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 2 = cluster1:2 ']' 2022-04-10T01:18:04.483 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:18:04.483 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:18:04.484 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=2 2022-04-10T01:18:04.484 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:18:04.484 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:18:04.484 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:18:04.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:18:04.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:18:04.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:2 cluster1 2022-04-10T01:18:04.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:2 2022-04-10T01:18:04.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:18:04.486 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:18:04.486 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2022-04-10T01:18:04.486 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:2 2022-04-10T01:18:04.486 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:18:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:18:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:18:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:18:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 2 = cluster1:2 ']' 2022-04-10T01:18:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:18:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:18:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=2 2022-04-10T01:18:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=2 2022-04-10T01:18:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:18:04.489 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:18:04.489 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:18:04.489 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:18:04.489 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2022-04-10T01:18:04.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:18:04.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:18:04.495 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:18:04.495 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2022-04-10T01:18:04.585 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/test'] 2022-04-10T01:18:04.585 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:18:04.585 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:18:04.589 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:18:04.590 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2022-04-10T01:18:04.590 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:3 2022-04-10T01:18:04.590 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:18:04.590 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:18:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:3 cluster instance 2022-04-10T01:18:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:3 2022-04-10T01:18:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:18:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:18:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:18:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=3 2022-04-10T01:18:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 3 = cluster1:3 ']' 2022-04-10T01:18:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:18:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:18:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=3 2022-04-10T01:18:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=3 2022-04-10T01:18:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:18:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:18:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:18:04.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:18:04.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:3 cluster1 2022-04-10T01:18:04.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:3 2022-04-10T01:18:04.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:18:04.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:18:04.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2022-04-10T01:18:04.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:3 2022-04-10T01:18:04.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:18:04.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:18:04.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:18:04.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=3 2022-04-10T01:18:04.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 3 = cluster1:3 ']' 2022-04-10T01:18:04.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:18:04.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:18:04.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=3 2022-04-10T01:18:04.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=3 2022-04-10T01:18:04.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:18:04.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:18:04.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:18:04.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:18:04.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2022-04-10T01:18:04.603 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:18:04.603 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:18:04.603 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:18:04.604 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2022-04-10T01:18:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/test'] 2022-04-10T01:18:04.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:18:04.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:18:04.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:18:04.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 4 2022-04-10T01:18:08.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:18:08.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:18:08.701 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:18:08.701 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2022-04-10T01:18:08.701 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:18:08.701 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:18:08.702 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:18:08.702 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:18:08.702 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:18:08.702 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:18:08.702 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:18:08.703 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:18:08.703 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:18:08.703 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:18:08.703 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:18:08.704 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:18:08.704 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:18:08.704 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:18:08.704 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:18:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:18:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:18:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:18:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:18:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:18:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:18:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:18:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:18:08.707 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:18:08.707 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:18:08.707 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:18:08.707 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:18:08.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:18:08.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:18:08.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:18:08.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:18:08.709 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:18:08.709 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:18:08.709 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:18:08.709 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:18:08.709 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:18:08.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:18:08.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:18:08.718 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:18:08.718 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:18:08.719 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:18:08.719 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2022-04-10T01:18:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/test'] 2022-04-10T01:18:08.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:18:08.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:18:08.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:18:08.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2022-04-10T01:18:08.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:1 2022-04-10T01:18:08.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:18:08.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:18:08.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:1 cluster instance 2022-04-10T01:18:08.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:1 2022-04-10T01:18:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:18:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:18:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:18:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=1 2022-04-10T01:18:08.824 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 1 = cluster1:1 ']' 2022-04-10T01:18:08.824 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:18:08.824 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:18:08.824 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=1 2022-04-10T01:18:08.824 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:18:08.825 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:18:08.825 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:18:08.825 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:18:08.825 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:18:08.825 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:1 cluster1 2022-04-10T01:18:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:1 2022-04-10T01:18:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:18:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:18:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2022-04-10T01:18:08.827 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:1 2022-04-10T01:18:08.827 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:18:08.827 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:18:08.827 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:18:08.827 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:18:08.828 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 1 = cluster1:1 ']' 2022-04-10T01:18:08.828 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:18:08.828 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:18:08.828 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=1 2022-04-10T01:18:08.828 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=1 2022-04-10T01:18:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:18:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:18:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:18:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:18:08.830 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2022-04-10T01:18:08.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:18:08.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:18:08.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:18:08.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2022-04-10T01:18:08.926 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/test'] 2022-04-10T01:18:08.927 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:18:08.927 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:18:08.931 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:18:08.931 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2022-04-10T01:18:08.932 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:2 2022-04-10T01:18:08.932 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:18:08.932 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:18:08.932 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:2 cluster instance 2022-04-10T01:18:08.933 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:2 2022-04-10T01:18:08.933 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:18:08.933 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:18:08.933 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:18:08.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=2 2022-04-10T01:18:08.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 2 = cluster1:2 ']' 2022-04-10T01:18:08.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:18:08.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:18:08.935 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=2 2022-04-10T01:18:08.935 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:18:08.935 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:18:08.935 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:18:08.935 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:18:08.936 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:18:08.936 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:2 cluster1 2022-04-10T01:18:08.936 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:2 2022-04-10T01:18:08.936 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:18:08.937 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:18:08.937 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2022-04-10T01:18:08.937 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:2 2022-04-10T01:18:08.937 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:18:08.937 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:18:08.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:18:08.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:18:08.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 2 = cluster1:2 ']' 2022-04-10T01:18:08.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:18:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:18:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=2 2022-04-10T01:18:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=2 2022-04-10T01:18:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:18:08.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:18:08.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:18:08.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:18:08.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2022-04-10T01:18:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:18:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:18:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:18:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2022-04-10T01:18:09.035 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/test'] 2022-04-10T01:18:09.036 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:18:09.036 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:18:09.040 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:18:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2022-04-10T01:18:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:3 2022-04-10T01:18:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:18:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:18:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:3 cluster instance 2022-04-10T01:18:09.042 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:3 2022-04-10T01:18:09.042 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:18:09.042 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:18:09.042 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:18:09.043 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=3 2022-04-10T01:18:09.043 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 3 = cluster1:3 ']' 2022-04-10T01:18:09.043 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:18:09.043 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:18:09.044 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=3 2022-04-10T01:18:09.044 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=3 2022-04-10T01:18:09.044 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:18:09.044 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:18:09.044 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:18:09.045 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:18:09.045 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:3 cluster1 2022-04-10T01:18:09.045 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:3 2022-04-10T01:18:09.045 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:18:09.046 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:18:09.046 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2022-04-10T01:18:09.046 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:3 2022-04-10T01:18:09.046 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:18:09.046 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:18:09.047 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:18:09.047 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=3 2022-04-10T01:18:09.047 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 3 = cluster1:3 ']' 2022-04-10T01:18:09.047 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:18:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:18:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=3 2022-04-10T01:18:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=3 2022-04-10T01:18:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:18:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:18:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:18:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:18:09.050 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2022-04-10T01:18:09.054 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:18:09.054 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:18:09.054 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:18:09.055 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2022-04-10T01:18:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/test'] 2022-04-10T01:18:09.145 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:18:09.145 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:18:09.146 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:18:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 8 2022-04-10T01:18:09.733 INFO:tasks.ceph.cluster2.osd.2.smithi043.stderr:2022-04-10T01:18:09.732+0000 7f8be0453700 -1 reset not still connected to 0x55f2a45be0d0 2022-04-10T01:18:09.733 INFO:tasks.ceph.cluster2.osd.2.smithi043.stderr:2022-04-10T01:18:09.732+0000 7f8be0453700 -1 reset not still connected to 0x55f2a45be4e0 2022-04-10T01:18:17.147 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:18:17.147 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:18:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:18:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2022-04-10T01:18:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:18:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:18:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:18:17.153 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:18:17.153 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:18:17.153 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:18:17.153 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:18:17.153 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:18:17.154 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:18:17.154 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:18:17.154 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:18:17.154 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:18:17.154 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:18:17.155 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:18:17.155 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:18:17.155 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:18:17.155 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:18:17.155 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:18:17.156 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:18:17.156 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:18:17.156 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:18:17.156 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:18:17.156 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:18:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:18:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:18:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:18:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:18:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:18:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:18:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:18:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:18:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:18:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:18:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:18:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:18:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:18:17.160 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:18:17.160 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:18:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:18:17.169 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:18:17.169 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:18:17.169 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2022-04-10T01:18:17.283 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:18:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:18:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:18:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:18:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:18:17.287 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:18:17.287 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:18:17.287 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:18:17.288 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ clean_snap_name= 2022-04-10T01:18:17.288 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ seq 1 10 2022-04-10T01:18:17.288 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 10` 2022-04-10T01:18:17.288 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror test 2022-04-10T01:18:17.289 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:18:17.289 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:18:17.289 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:18:17.289 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:18:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=11 2022-04-10T01:18:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:18:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:18:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:18:17.292 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:18:17.292 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:18:17.292 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2022-04-10T01:18:17.436 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 53612544 bytes 2022-04-10T01:18:17.568 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 140863488 bytes 2022-04-10T01:18:17.648 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 174067712 bytes 2022-04-10T01:18:17.723 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 205565952 bytes 2022-04-10T01:18:18.004 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 379074560 bytes 2022-04-10T01:18:18.051 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 388268032 bytes 2022-04-10T01:18:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 423319552 bytes 2022-04-10T01:18:18.182 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 428171264 bytes 2022-04-10T01:18:18.223 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 428803072 bytes 2022-04-10T01:18:18.302 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 478425088 bytes 2022-04-10T01:18:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 514365440 bytes 2022-04-10T01:18:18.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 592497664 bytes 2022-04-10T01:18:18.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 603640832 bytes 2022-04-10T01:18:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 651947008 bytes 2022-04-10T01:18:18.977 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 681352192 bytes 2022-04-10T01:18:19.123 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 769780736 bytes 2022-04-10T01:18:19.197 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 794254336 bytes 2022-04-10T01:18:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 897480704 bytes 2022-04-10T01:18:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 964556800 bytes 2022-04-10T01:18:19.758 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 997965824 bytes 2022-04-10T01:18:19.830 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 1021960192 bytes 2022-04-10T01:18:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 1062268928 bytes 2022-04-10T01:18:19.996 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 1091609600 bytes 2022-04-10T01:18:20.087 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 1120609280 bytes 2022-04-10T01:18:20.184 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 1164235776 bytes 2022-04-10T01:18:20.314 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1238672384 bytes 2022-04-10T01:18:20.418 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1287712768 bytes 2022-04-10T01:18:20.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1329927168 bytes 2022-04-10T01:18:20.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1403291648 bytes 2022-04-10T01:18:20.760 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1440446464 bytes 2022-04-10T01:18:20.885 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1536657408 bytes 2022-04-10T01:18:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1609064448 bytes 2022-04-10T01:18:21.111 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1609702400 bytes 2022-04-10T01:18:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1634978816 bytes 2022-04-10T01:18:21.361 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1744179200 bytes 2022-04-10T01:18:21.472 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1790758912 bytes 2022-04-10T01:18:21.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1828488192 bytes 2022-04-10T01:18:21.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1891495936 bytes 2022-04-10T01:18:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1966306304 bytes 2022-04-10T01:18:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1981196288 bytes 2022-04-10T01:18:21.985 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 2040331264 bytes 2022-04-10T01:18:22.073 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 2055212032 bytes 2022-04-10T01:18:22.172 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 2104658944 bytes 2022-04-10T01:18:22.267 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 2141641728 bytes 2022-04-10T01:18:22.388 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 2196561920 bytes 2022-04-10T01:18:22.492 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 2238489600 bytes 2022-04-10T01:18:22.616 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 2304352256 bytes 2022-04-10T01:18:22.747 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 2386171904 bytes 2022-04-10T01:18:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 2424330240 bytes 2022-04-10T01:18:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 2458638336 bytes 2022-04-10T01:18:23.029 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 2487451648 bytes 2022-04-10T01:18:23.107 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 2512413696 bytes 2022-04-10T01:18:23.155 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 2513034240 bytes 2022-04-10T01:18:23.280 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5400 IOs, 2584580096 bytes 2022-04-10T01:18:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5500 IOs, 2665225216 bytes 2022-04-10T01:18:23.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5600 IOs, 2695705600 bytes 2022-04-10T01:18:23.618 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5700 IOs, 2729470976 bytes 2022-04-10T01:18:23.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5800 IOs, 2731509760 bytes 2022-04-10T01:18:23.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5900 IOs, 2808530944 bytes 2022-04-10T01:18:23.900 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6000 IOs, 2847159296 bytes 2022-04-10T01:18:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6100 IOs, 2859138048 bytes 2022-04-10T01:18:24.102 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6200 IOs, 2933575680 bytes 2022-04-10T01:18:24.200 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6300 IOs, 2971933696 bytes 2022-04-10T01:18:24.304 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6400 IOs, 3021307904 bytes 2022-04-10T01:18:24.405 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6500 IOs, 3069730816 bytes 2022-04-10T01:18:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6600 IOs, 3102967808 bytes 2022-04-10T01:18:24.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6700 IOs, 3127067648 bytes 2022-04-10T01:18:24.712 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6800 IOs, 3182496768 bytes 2022-04-10T01:18:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6900 IOs, 3193470976 bytes 2022-04-10T01:18:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7000 IOs, 3218449408 bytes 2022-04-10T01:18:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7100 IOs, 3227090944 bytes 2022-04-10T01:18:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7200 IOs, 3250946048 bytes 2022-04-10T01:18:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7300 IOs, 3296508928 bytes 2022-04-10T01:18:25.245 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7400 IOs, 3390936064 bytes 2022-04-10T01:18:25.328 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7500 IOs, 3403028480 bytes 2022-04-10T01:18:25.402 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7600 IOs, 3418517504 bytes 2022-04-10T01:18:25.454 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7700 IOs, 3419105280 bytes 2022-04-10T01:18:25.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7800 IOs, 3448944640 bytes 2022-04-10T01:18:25.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7900 IOs, 3456778240 bytes 2022-04-10T01:18:25.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8000 IOs, 3495064576 bytes 2022-04-10T01:18:25.775 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8100 IOs, 3516849152 bytes 2022-04-10T01:18:25.911 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8200 IOs, 3588395008 bytes 2022-04-10T01:18:26.006 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8300 IOs, 3624905728 bytes 2022-04-10T01:18:26.113 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8400 IOs, 3673511936 bytes 2022-04-10T01:18:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8500 IOs, 3727798272 bytes 2022-04-10T01:18:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8600 IOs, 3786958848 bytes 2022-04-10T01:18:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8700 IOs, 3832504320 bytes 2022-04-10T01:18:26.629 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8800 IOs, 3916687360 bytes 2022-04-10T01:18:26.744 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8900 IOs, 3963037696 bytes 2022-04-10T01:18:26.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:9000 IOs, 3981465600 bytes 2022-04-10T01:18:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:9100 IOs, 4020129792 bytes 2022-04-10T01:18:26.973 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:9200 IOs, 4065244160 bytes 2022-04-10T01:18:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:9300 IOs, 4089071616 bytes 2022-04-10T01:18:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:9400 IOs, 4173405184 bytes 2022-04-10T01:18:27.329 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:9500 IOs, 4192702464 bytes 2022-04-10T01:18:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:9600 IOs, 4253310976 bytes 2022-04-10T01:18:27.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:9700 IOs, 4285078528 bytes 2022-04-10T01:18:27.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:9800 IOs, 4312958976 bytes 2022-04-10T01:18:27.758 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3539012265967325) > weight (0.25) 2022-04-10T01:18:27.758 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.29585149690953805) > weight (0.25) 2022-04-10T01:18:27.759 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.3723713553030895) > weight (0.25) 2022-04-10T01:18:27.759 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2022-04-10T01:18:27.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2022-04-10T01:18:27.761 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2022-04-10T01:18:27.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr:2022-04-10T01:18:27.760+0000 7fae133e4700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 39557) UID: 1000 2022-04-10T01:18:27.769 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:9900 IOs, 4370041856 bytes 2022-04-10T01:18:27.840 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:10000 IOs, 4390470656 bytes 2022-04-10T01:18:27.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr:daemon-helper: command failed with exit status 1 2022-04-10T01:18:27.962 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:10100 IOs, 4436616192 bytes 2022-04-10T01:18:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:10200 IOs, 4527201280 bytes 2022-04-10T01:18:28.297 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:18:28.297 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:18:28.297 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:18:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:18:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2022-04-10T01:18:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:18:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:18:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:18:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state_pattern=up+replaying 2022-04-10T01:18:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local description_pattern= 2022-04-10T01:18:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local service_pattern= 2022-04-10T01:18:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:18:28.300 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:18:28.300 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:18:28.301 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:18:28.301 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:18:28.301 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:18:29.300 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2022-04-10T01:18:29.300 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:18:29.300 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:18:29.300 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:18:29.301 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state_pattern=up+replaying 2022-04-10T01:18:29.301 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local description_pattern= 2022-04-10T01:18:29.301 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local service_pattern= 2022-04-10T01:18:29.301 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:18:29.301 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:18:29.302 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:18:29.302 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:18:29.302 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-test.mirror_status 2022-04-10T01:18:29.302 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-test.mirror_status 2022-04-10T01:18:29.303 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:18:29.303 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:18:29.304 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ CEPH_ARGS= 2022-04-10T01:18:29.304 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2022-04-10T01:18:29.304 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ tee /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:18:29.344 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:test: 2022-04-10T01:18:29.344 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: global_id: 5ff848f4-a6a0-498d-b9f1-7a041de7de98 2022-04-10T01:18:29.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: state: down+unknown 2022-04-10T01:18:29.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: description: status not found 2022-04-10T01:18:29.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: last_update: 2022-04-10T01:18:29.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: peer_sites: 2022-04-10T01:18:29.346 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: name: cluster2 2022-04-10T01:18:29.346 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: state: up+stopped 2022-04-10T01:18:29.346 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: description: local image is primary 2022-04-10T01:18:29.346 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: last_update: 2022-04-10 01:18:11 2022-04-10T01:18:29.349 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:18:29.350 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 1 2022-04-10T01:18:29.351 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:18:29.351 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 2 2022-04-10T01:18:30.762 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2022-04-10T01:18:30.763 INFO:teuthology.orchestra.run:waiting for 600 2022-04-10T01:18:30.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-04-10T01:18:30.764 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2022-04-10T01:18:30.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2022-04-10T01:18:30.764 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2022-04-10T01:18:30.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2022-04-10T01:18:30.768 INFO:tasks.rbd_mirror_thrash:waiting for 102 secs before thrashing 2022-04-10T01:18:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2022-04-10T01:18:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:18:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:18:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:18:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state_pattern=up+replaying 2022-04-10T01:18:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local description_pattern= 2022-04-10T01:18:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local service_pattern= 2022-04-10T01:18:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:18:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:18:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:18:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:18:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-test.mirror_status 2022-04-10T01:18:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-test.mirror_status 2022-04-10T01:18:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:18:31.357 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:18:31.357 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ CEPH_ARGS= 2022-04-10T01:18:31.358 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2022-04-10T01:18:31.358 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ tee /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:18:31.400 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:test: 2022-04-10T01:18:31.400 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: global_id: 5ff848f4-a6a0-498d-b9f1-7a041de7de98 2022-04-10T01:18:31.400 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: state: down+unknown 2022-04-10T01:18:31.401 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: description: status not found 2022-04-10T01:18:31.401 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: last_update: 2022-04-10T01:18:31.401 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: peer_sites: 2022-04-10T01:18:31.401 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: name: cluster2 2022-04-10T01:18:31.402 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: state: up+stopped 2022-04-10T01:18:31.402 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: description: local image is primary 2022-04-10T01:18:31.402 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: last_update: 2022-04-10 01:18:11 2022-04-10T01:18:31.404 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:18:31.406 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 1 2022-04-10T01:18:31.406 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:18:31.406 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 4 2022-04-10T01:18:33.758 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8359320795850216) > weight (0.25) 2022-04-10T01:18:33.759 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5975763802088061) > weight (0.25) 2022-04-10T01:18:33.759 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.387034268757361) > weight (0.25) 2022-04-10T01:18:33.759 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.9977505331564969) > weight (0.25) 2022-04-10T01:18:33.760 INFO:tasks.rbd_mirror_thrash:waiting for 95 secs before thrashing 2022-04-10T01:18:35.408 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2022-04-10T01:18:35.408 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:18:35.409 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:18:35.409 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:18:35.409 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state_pattern=up+replaying 2022-04-10T01:18:35.409 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local description_pattern= 2022-04-10T01:18:35.410 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local service_pattern= 2022-04-10T01:18:35.410 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:18:35.410 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:18:35.410 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:18:35.411 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:18:35.411 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-test.mirror_status 2022-04-10T01:18:35.411 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-test.mirror_status 2022-04-10T01:18:35.411 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:18:35.412 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:18:35.412 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ CEPH_ARGS= 2022-04-10T01:18:35.412 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2022-04-10T01:18:35.413 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ tee /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:18:35.454 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:test: 2022-04-10T01:18:35.454 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: global_id: 5ff848f4-a6a0-498d-b9f1-7a041de7de98 2022-04-10T01:18:35.454 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: state: down+unknown 2022-04-10T01:18:35.455 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: description: status not found 2022-04-10T01:18:35.455 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: last_update: 2022-04-10T01:18:35.455 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: peer_sites: 2022-04-10T01:18:35.455 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: name: cluster2 2022-04-10T01:18:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: state: up+stopped 2022-04-10T01:18:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: description: local image is primary 2022-04-10T01:18:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: last_update: 2022-04-10 01:18:11 2022-04-10T01:18:35.459 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:18:35.460 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 1 2022-04-10T01:18:35.460 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:18:35.461 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 8 2022-04-10T01:18:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2022-04-10T01:18:43.463 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:18:43.463 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:18:43.463 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:18:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state_pattern=up+replaying 2022-04-10T01:18:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local description_pattern= 2022-04-10T01:18:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local service_pattern= 2022-04-10T01:18:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:18:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:18:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:18:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:18:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-test.mirror_status 2022-04-10T01:18:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-test.mirror_status 2022-04-10T01:18:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:18:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:18:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ CEPH_ARGS= 2022-04-10T01:18:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2022-04-10T01:18:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ tee /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:18:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:test: 2022-04-10T01:18:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: global_id: 5ff848f4-a6a0-498d-b9f1-7a041de7de98 2022-04-10T01:18:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: state: up+replaying 2022-04-10T01:18:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1649553479,"remote_snapshot_timestamp":1649553479,"replay_state":"idle"} 2022-04-10T01:18:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: service: mirror.0 on smithi043 2022-04-10T01:18:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: last_update: 2022-04-10 01:18:39 2022-04-10T01:18:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: peer_sites: 2022-04-10T01:18:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: name: cluster2 2022-04-10T01:18:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: state: up+stopped 2022-04-10T01:18:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: description: local image is primary 2022-04-10T01:18:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: last_update: 2022-04-10 01:18:39 2022-04-10T01:18:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:18:43.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: state: up+replaying 2022-04-10T01:18:43.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep '^ description: .*' /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:18:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1649553479,"remote_snapshot_timestamp":1649553479,"replay_state":"idle"} 2022-04-10T01:18:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:18:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo up+replaying 2022-04-10T01:18:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep '^up+' 2022-04-10T01:18:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:up+replaying 2022-04-10T01:18:43.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:18:43.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: service: mirror.0 on smithi043 2022-04-10T01:18:43.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:18:43.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local 'last_update=2022-04-10 01:18:39' 2022-04-10T01:18:43.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2022-04-10 01:18:39' 2022-04-10T01:18:43.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:18:43.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:18:43.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:18:43.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state_pattern=up+replaying 2022-04-10T01:18:43.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local 'prev_last_update=2022-04-10 01:18:39' 2022-04-10T01:18:43.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:18:43.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:18:43.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:18:43.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:18:43.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror.mirror_status 2022-04-10T01:18:43.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror.mirror_status 2022-04-10T01:18:43.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:18:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror.mirror_status 2022-04-10T01:18:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2022-04-10T01:18:43.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: failed to query services: (13) Permission denied 2022-04-10T01:18:43.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local last_update state 2022-04-10T01:18:43.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2022-04-10T01:18:43.592 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ last_update='2022-04-10 01:18:39' 2022-04-10T01:18:43.592 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2022-04-10T01:18:43.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ state=up+replaying 2022-04-10T01:18:43.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo up+replaying 2022-04-10T01:18:43.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep up+replaying 2022-04-10T01:18:43.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:up+replaying 2022-04-10T01:18:43.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:18:43.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:18:43.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ snap_name=snap1 2022-04-10T01:18:43.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror test snap1 2022-04-10T01:18:43.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:18:43.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:18:43.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:18:43.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap1 2022-04-10T01:18:43.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:18:59.184 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror test 2022-04-10T01:18:59.185 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:18:59.185 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:18:59.185 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:18:59.186 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:18:59.186 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:18:59.186 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:18:59.187 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:18:59.187 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2022-04-10T01:18:59.187 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:18:59.187 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:18:59.188 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:18:59.188 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:18:59.188 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:18:59.188 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:18:59.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:18:59.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:18:59.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:18:59.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:18:59.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:19:00.186 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror test started 2022-04-10T01:19:00.186 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:19:00.187 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:19:00.187 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:19:00.187 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:19:00.187 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:19:00.188 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:19:00.188 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:19:00.188 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:19:00.188 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:19:00.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:19:00.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2022-04-10T01:19:00.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:19:00.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:19:00.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:19:00.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:19:00.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:19:00.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:19:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:19:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:19:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:19:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:19:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:19:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:19:00.196 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:19:00.196 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:19:00.196 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:19:00.197 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2022-04-10T01:19:00.197 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:19:00.197 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:19:00.197 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:19:00.198 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:19:00.198 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:19:00.198 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:19:00.198 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:19:00.198 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:19:00.199 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:19:00.199 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:19:00.199 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:19:00.199 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:19:00.200 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:19:00.200 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:19:00.200 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:19:00.200 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:19:00.200 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:19:00.201 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:19:00.201 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:19:00.201 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:19:00.201 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:19:00.202 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:19:00.202 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:19:00.202 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:19:00.202 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:19:00.202 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:19:00.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:19:00.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:19:00.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:19:00.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:19:00.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:19:00.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:19:00.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:19:00.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:19:00.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:19:00.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:19:00.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:19:00.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:19:00.208 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:19:00.208 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:19:00.209 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:19:00.209 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2022-04-10T01:19:00.320 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:19:00.321 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:19:00.322 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:19:00.322 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:19:00.323 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:19:00.327 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:19:00.327 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:19:00.327 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:19:00.328 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2022-04-10T01:19:00.328 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:19:00.328 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:19:00.328 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:19:00.329 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:19:00.329 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:19:00.329 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:19:00.329 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:19:00.329 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:19:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:19:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:19:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2022-04-10T01:19:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:19:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:19:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:19:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:19:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:19:00.332 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:19:00.332 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:19:00.332 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:19:00.332 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:19:00.332 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:19:00.333 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:19:00.333 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:19:00.333 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-test.status 2022-04-10T01:19:00.333 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-test.status 2022-04-10T01:19:00.334 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:19:00.334 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.status 2022-04-10T01:19:00.334 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-test.status 2022-04-10T01:19:00.334 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-test.status 2022-04-10T01:19:00.334 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:19:00.335 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:19:00.335 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror test 2022-04-10T01:19:00.335 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:19:00.335 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:19:00.336 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:19:00.336 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:19:00.336 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:19:00.336 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:19:00.336 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:19:00.337 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2022-04-10T01:19:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 6 2022-04-10T01:19:01.196 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.rgwz74WspX/cluster2-mirror-test.status 2022-04-10T01:19:01.197 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:19:01.197 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:19:01.197 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:19:01.197 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.status 2022-04-10T01:19:01.198 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:19:01.198 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:19:01.198 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:19:01.198 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:19:01.199 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2022-04-10T01:19:01.199 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:19:01.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:19:01.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=6 2022-04-10T01:19:01.242 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:19:01.242 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:19:01.242 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:19:01.443 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:19:01.444 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:19:01.444 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:19:01.444 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:19:01.444 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:19:01.444 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:19:01.445 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:19:01.445 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:19:01.445 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:19:01.445 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2022-04-10T01:19:01.446 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:19:01.490 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:19:01.492 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=4 2022-04-10T01:19:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 6 = 4 2022-04-10T01:19:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:19:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:19:01.894 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:19:01.894 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:19:01.895 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:19:01.895 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:19:01.895 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:19:01.895 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:19:01.896 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:19:01.896 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:19:01.896 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:19:01.896 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2022-04-10T01:19:01.897 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:19:01.942 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:19:01.942 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:19:01.943 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:19:01.944 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:19:01.944 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:19:01.945 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:19:01.945 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 6 = '' 2022-04-10T01:19:01.945 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:19:01.946 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:19:02.746 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:19:02.747 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:19:02.747 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:19:02.747 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:19:02.747 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:19:02.748 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:19:02.748 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:19:02.748 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:19:02.748 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:19:02.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2022-04-10T01:19:02.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:19:02.810 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:19:02.811 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:19:02.812 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:19:02.812 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:19:02.812 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:19:02.813 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:19:02.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 6 = '' 2022-04-10T01:19:02.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:19:02.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1.6 2022-04-10T01:19:04.415 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:19:04.416 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:19:04.416 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:19:04.416 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:19:04.416 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:19:04.417 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:19:04.417 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:19:04.417 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:19:04.417 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:19:04.418 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2022-04-10T01:19:04.418 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:19:04.464 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:19:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=6 2022-04-10T01:19:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 6 = 6 2022-04-10T01:19:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:19:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror test snap1 2022-04-10T01:19:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:19:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:19:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:19:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap1 2022-04-10T01:19:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:19:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:19:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:19:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:19:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:19:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:19:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:19:05.467 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/test@snap1 2022-04-10T01:19:05.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'test': 2022-04-10T01:19:05.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 512 MiB in 128 objects 2022-04-10T01:19:05.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:19:05.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 2 2022-04-10T01:19:05.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 10ae4a893004 2022-04-10T01:19:05.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.10ae4a893004 2022-04-10T01:19:05.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:19:05.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:19:05.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:19:05.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:19:05.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:18:11 2022 2022-04-10T01:19:05.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:18:11 2022 2022-04-10T01:19:05.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:18:11 2022 2022-04-10T01:19:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:19:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:19:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:19:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 5ff848f4-a6a0-498d-b9f1-7a041de7de98 2022-04-10T01:19:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:19:05.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:19:05.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:19:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror test snap1 2022-04-10T01:19:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:19:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:19:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap1 2022-04-10T01:19:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:19:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:19:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:19:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:19:05.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:19:06.110 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:19:06.114 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:19:06.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:19:06.879 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:19:07.129 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:19:07.273 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:19:07.273 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ clean_snap_name=snap1-clean 2022-04-10T01:19:07.273 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror test snap1-clean 2022-04-10T01:19:07.274 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:19:07.274 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:19:07.274 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:19:07.274 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap1-clean 2022-04-10T01:19:07.275 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1-clean --debug-rbd=20 --debug-journaler=20 2022-04-10T01:19:08.163 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 10` 2022-04-10T01:19:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror test 2022-04-10T01:19:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:19:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:19:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:19:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:19:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=13 2022-04-10T01:19:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:19:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:19:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:19:08.167 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:19:08.167 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:19:08.167 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2022-04-10T01:19:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 81618944 bytes 2022-04-10T01:19:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 152553472 bytes 2022-04-10T01:19:08.766 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 256201728 bytes 2022-04-10T01:19:08.897 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 289270784 bytes 2022-04-10T01:19:09.005 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 337894400 bytes 2022-04-10T01:19:09.076 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 343017472 bytes 2022-04-10T01:19:09.134 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 343609344 bytes 2022-04-10T01:19:09.263 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 406268928 bytes 2022-04-10T01:19:09.344 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 427567104 bytes 2022-04-10T01:19:09.437 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 458616832 bytes 2022-04-10T01:19:09.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 471775232 bytes 2022-04-10T01:19:09.648 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 533308416 bytes 2022-04-10T01:19:09.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 533943296 bytes 2022-04-10T01:19:09.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 578913280 bytes 2022-04-10T01:19:09.932 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 618897408 bytes 2022-04-10T01:19:10.112 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 726946816 bytes 2022-04-10T01:19:10.217 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 760922112 bytes 2022-04-10T01:19:10.282 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 769144832 bytes 2022-04-10T01:19:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 847296512 bytes 2022-04-10T01:19:10.592 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 934431744 bytes 2022-04-10T01:19:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 954689536 bytes 2022-04-10T01:19:10.778 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 987727872 bytes 2022-04-10T01:19:10.841 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 1001100288 bytes 2022-04-10T01:19:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 1047341056 bytes 2022-04-10T01:19:11.043 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 1063564288 bytes 2022-04-10T01:19:11.167 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1126831104 bytes 2022-04-10T01:19:11.234 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1127433216 bytes 2022-04-10T01:19:11.375 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1192999936 bytes 2022-04-10T01:19:11.493 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1240317952 bytes 2022-04-10T01:19:11.590 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1271113728 bytes 2022-04-10T01:19:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1288118272 bytes 2022-04-10T01:19:11.789 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1328222208 bytes 2022-04-10T01:19:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1351525376 bytes 2022-04-10T01:19:11.970 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1379274752 bytes 2022-04-10T01:19:12.054 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1401191424 bytes 2022-04-10T01:19:12.222 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1481681920 bytes 2022-04-10T01:19:12.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1538397184 bytes 2022-04-10T01:19:12.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1562500096 bytes 2022-04-10T01:19:12.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1592585216 bytes 2022-04-10T01:19:12.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1615818752 bytes 2022-04-10T01:19:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1651713024 bytes 2022-04-10T01:19:12.828 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 1689385984 bytes 2022-04-10T01:19:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 1734874112 bytes 2022-04-10T01:19:13.060 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 1771164672 bytes 2022-04-10T01:19:13.144 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 1791548416 bytes 2022-04-10T01:19:13.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 1818561536 bytes 2022-04-10T01:19:13.314 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 1837753344 bytes 2022-04-10T01:19:13.380 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 1838344192 bytes 2022-04-10T01:19:13.470 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 1871825920 bytes 2022-04-10T01:19:13.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 1950472192 bytes 2022-04-10T01:19:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 1964327936 bytes 2022-04-10T01:19:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 2002361344 bytes 2022-04-10T01:19:13.949 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 2050798592 bytes 2022-04-10T01:19:14.039 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5400 IOs, 2087158784 bytes 2022-04-10T01:19:14.179 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5500 IOs, 2141671424 bytes 2022-04-10T01:19:14.291 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5600 IOs, 2170909696 bytes 2022-04-10T01:19:14.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5700 IOs, 2237173760 bytes 2022-04-10T01:19:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5800 IOs, 2281553920 bytes 2022-04-10T01:19:14.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5900 IOs, 2301559808 bytes 2022-04-10T01:19:14.764 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6000 IOs, 2306597888 bytes 2022-04-10T01:19:14.862 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6100 IOs, 2340679680 bytes 2022-04-10T01:19:14.968 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6200 IOs, 2374348800 bytes 2022-04-10T01:19:15.201 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6300 IOs, 2453786624 bytes 2022-04-10T01:19:15.414 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6400 IOs, 2526947328 bytes 2022-04-10T01:19:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6500 IOs, 2550006784 bytes 2022-04-10T01:19:15.605 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6600 IOs, 2576664576 bytes 2022-04-10T01:19:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6700 IOs, 2616303616 bytes 2022-04-10T01:19:15.841 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6800 IOs, 2657782784 bytes 2022-04-10T01:19:15.968 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6900 IOs, 2707513344 bytes 2022-04-10T01:19:16.051 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7000 IOs, 2724960256 bytes 2022-04-10T01:19:16.215 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7100 IOs, 2805620736 bytes 2022-04-10T01:19:16.319 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7200 IOs, 2839774208 bytes 2022-04-10T01:19:16.396 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7300 IOs, 2851172352 bytes 2022-04-10T01:19:16.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7400 IOs, 2922476544 bytes 2022-04-10T01:19:16.616 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7500 IOs, 2933629952 bytes 2022-04-10T01:19:16.700 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7600 IOs, 2950448128 bytes 2022-04-10T01:19:16.788 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7700 IOs, 2963835904 bytes 2022-04-10T01:19:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7800 IOs, 3039295488 bytes 2022-04-10T01:19:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7900 IOs, 3055204352 bytes 2022-04-10T01:19:17.167 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8000 IOs, 3126007808 bytes 2022-04-10T01:19:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8100 IOs, 3130436608 bytes 2022-04-10T01:19:17.354 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8200 IOs, 3171425280 bytes 2022-04-10T01:19:17.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8300 IOs, 3260483584 bytes 2022-04-10T01:19:17.605 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8400 IOs, 3277235200 bytes 2022-04-10T01:19:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8500 IOs, 3298191360 bytes 2022-04-10T01:19:17.840 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8600 IOs, 3364827136 bytes 2022-04-10T01:19:17.912 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8700 IOs, 3371230208 bytes 2022-04-10T01:19:18.022 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8800 IOs, 3407916032 bytes 2022-04-10T01:19:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8900 IOs, 3447153664 bytes 2022-04-10T01:19:18.266 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:9000 IOs, 3501086720 bytes 2022-04-10T01:19:18.393 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:9100 IOs, 3550431232 bytes 2022-04-10T01:19:18.467 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:9200 IOs, 3557122048 bytes 2022-04-10T01:19:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:9300 IOs, 3635301376 bytes 2022-04-10T01:19:18.704 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:9400 IOs, 3645669376 bytes 2022-04-10T01:19:18.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:9500 IOs, 3666144256 bytes 2022-04-10T01:19:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:9600 IOs, 3690675200 bytes 2022-04-10T01:19:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:9700 IOs, 3731784704 bytes 2022-04-10T01:19:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:9800 IOs, 3806149632 bytes 2022-04-10T01:19:19.299 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:9900 IOs, 3857617920 bytes 2022-04-10T01:19:19.438 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:10000 IOs, 3917540352 bytes 2022-04-10T01:19:19.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:10100 IOs, 3924325376 bytes 2022-04-10T01:19:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:10200 IOs, 3966032896 bytes 2022-04-10T01:19:19.716 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:10300 IOs, 3981124608 bytes 2022-04-10T01:19:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:10400 IOs, 4064854016 bytes 2022-04-10T01:19:20.007 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:10500 IOs, 4105546752 bytes 2022-04-10T01:19:20.159 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:10600 IOs, 4186534912 bytes 2022-04-10T01:19:20.275 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:10700 IOs, 4218866688 bytes 2022-04-10T01:19:20.467 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:10800 IOs, 4326567936 bytes 2022-04-10T01:19:20.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:10900 IOs, 4327207936 bytes 2022-04-10T01:19:20.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:11000 IOs, 4397237248 bytes 2022-04-10T01:19:20.811 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:11100 IOs, 4430399488 bytes 2022-04-10T01:19:20.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:11200 IOs, 4473713664 bytes 2022-04-10T01:19:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:11300 IOs, 4514220032 bytes 2022-04-10T01:19:21.180 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:19:21.180 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:19:21.180 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:19:21.181 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:19:21.181 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2022-04-10T01:19:21.181 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:19:21.181 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:19:21.182 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:19:21.182 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state_pattern=up+replaying 2022-04-10T01:19:21.182 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local description_pattern= 2022-04-10T01:19:21.182 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local service_pattern= 2022-04-10T01:19:21.182 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:19:21.183 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:19:21.183 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:19:21.184 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:19:21.184 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:19:21.184 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:19:22.182 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2022-04-10T01:19:22.182 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:19:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:19:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:19:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state_pattern=up+replaying 2022-04-10T01:19:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local description_pattern= 2022-04-10T01:19:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local service_pattern= 2022-04-10T01:19:22.184 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:19:22.184 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:19:22.184 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:19:22.184 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:19:22.185 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-test.mirror_status 2022-04-10T01:19:22.185 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-test.mirror_status 2022-04-10T01:19:22.185 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:19:22.186 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:19:22.186 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ CEPH_ARGS= 2022-04-10T01:19:22.186 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2022-04-10T01:19:22.187 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ tee /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:19:22.231 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:test: 2022-04-10T01:19:22.231 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: global_id: 5ff848f4-a6a0-498d-b9f1-7a041de7de98 2022-04-10T01:19:22.231 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: state: up+replaying 2022-04-10T01:19:22.231 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: description: replaying, {"bytes_per_second":17882794.67,"bytes_per_snapshot":268241920.0,"local_snapshot_timestamp":1649553541,"remote_snapshot_timestamp":1649553541,"replay_state":"idle"} 2022-04-10T01:19:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: service: mirror.0 on smithi043 2022-04-10T01:19:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: last_update: 2022-04-10 01:19:09 2022-04-10T01:19:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: peer_sites: 2022-04-10T01:19:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: name: cluster2 2022-04-10T01:19:22.233 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: state: up+stopped 2022-04-10T01:19:22.233 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: description: local image is primary 2022-04-10T01:19:22.233 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: last_update: 2022-04-10 01:19:09 2022-04-10T01:19:22.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:19:22.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: state: up+replaying 2022-04-10T01:19:22.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep '^ description: .*' /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:19:22.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: description: replaying, {"bytes_per_second":17882794.67,"bytes_per_snapshot":268241920.0,"local_snapshot_timestamp":1649553541,"remote_snapshot_timestamp":1649553541,"replay_state":"idle"} 2022-04-10T01:19:22.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:19:22.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo up+replaying 2022-04-10T01:19:22.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep '^up+' 2022-04-10T01:19:22.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:up+replaying 2022-04-10T01:19:22.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:19:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: service: mirror.0 on smithi043 2022-04-10T01:19:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:19:22.243 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local 'last_update=2022-04-10 01:19:09' 2022-04-10T01:19:22.243 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2022-04-10 01:19:09' 2022-04-10T01:19:22.243 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:19:22.244 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:19:22.244 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:19:22.244 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state_pattern=up+replaying 2022-04-10T01:19:22.244 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local 'prev_last_update=2022-04-10 01:19:09' 2022-04-10T01:19:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:19:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:19:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:19:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:19:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror.mirror_status 2022-04-10T01:19:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror.mirror_status 2022-04-10T01:19:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:19:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror.mirror_status 2022-04-10T01:19:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2022-04-10T01:19:22.279 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: failed to query services: (13) Permission denied 2022-04-10T01:19:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local last_update state 2022-04-10T01:19:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2022-04-10T01:19:22.303 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ last_update='2022-04-10 01:19:09' 2022-04-10T01:19:22.303 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2022-04-10T01:19:22.305 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ state=up+replaying 2022-04-10T01:19:22.305 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo up+replaying 2022-04-10T01:19:22.306 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep up+replaying 2022-04-10T01:19:22.306 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:up+replaying 2022-04-10T01:19:22.307 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:19:22.308 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:19:22.308 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ snap_name=snap2 2022-04-10T01:19:22.308 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror test snap2 2022-04-10T01:19:22.308 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:19:22.309 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:19:22.309 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:19:22.309 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap2 2022-04-10T01:19:22.309 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:19:52.183 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror test 2022-04-10T01:19:52.183 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:19:52.184 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:19:52.184 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:19:52.184 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:19:52.184 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:19:52.185 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:19:52.185 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:19:52.185 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2022-04-10T01:19:52.185 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:19:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:19:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:19:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:19:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:19:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:19:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:19:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:19:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:19:52.188 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:19:52.188 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:19:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror test started 2022-04-10T01:19:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:19:53.186 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:19:53.186 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:19:53.186 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:19:53.186 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:19:53.187 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:19:53.187 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:19:53.187 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:19:53.187 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:19:53.187 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:19:53.188 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2022-04-10T01:19:53.188 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:19:53.188 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:19:53.188 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:19:53.188 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:19:53.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:19:53.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:19:53.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:19:53.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:19:53.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:19:53.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:19:53.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:19:53.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:19:53.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:19:53.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:19:53.195 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:19:53.195 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2022-04-10T01:19:53.195 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:19:53.195 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:19:53.196 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:19:53.196 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:19:53.196 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:19:53.196 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:19:53.196 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:19:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:19:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:19:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:19:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:19:53.198 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:19:53.198 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:19:53.198 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:19:53.198 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:19:53.199 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:19:53.199 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:19:53.199 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:19:53.199 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:19:53.199 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:19:53.200 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:19:53.200 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:19:53.200 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:19:53.200 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:19:53.200 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:19:53.201 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:19:53.201 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:19:53.201 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:19:53.201 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:19:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:19:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:19:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:19:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:19:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:19:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:19:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:19:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:19:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:19:53.207 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:19:53.207 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:19:53.207 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:19:53.207 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2022-04-10T01:19:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:19:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:19:53.326 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:19:53.326 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:19:53.327 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:19:53.336 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:19:53.336 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:19:53.336 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:19:53.336 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2022-04-10T01:19:53.337 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:19:53.337 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:19:53.337 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:19:53.337 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:19:53.338 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:19:53.338 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:19:53.338 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:19:53.338 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:19:53.339 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:19:53.339 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:19:53.339 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2022-04-10T01:19:53.339 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:19:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:19:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:19:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:19:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:19:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:19:53.341 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:19:53.341 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:19:53.341 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:19:53.341 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:19:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:19:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:19:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-test.status 2022-04-10T01:19:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-test.status 2022-04-10T01:19:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:19:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.status 2022-04-10T01:19:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-test.status 2022-04-10T01:19:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-test.status 2022-04-10T01:19:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:19:53.344 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:19:53.344 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror test 2022-04-10T01:19:53.344 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:19:53.344 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:19:53.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:19:53.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:19:53.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:19:53.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:19:53.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:19:53.346 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2022-04-10T01:19:54.186 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 10 2022-04-10T01:19:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.rgwz74WspX/cluster2-mirror-test.status 2022-04-10T01:19:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:19:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:19:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:19:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.status 2022-04-10T01:19:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:19:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:19:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:19:54.198 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:19:54.198 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2022-04-10T01:19:54.198 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:19:54.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:19:54.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=10 2022-04-10T01:19:54.242 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:19:54.242 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:19:54.242 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:19:54.443 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:19:54.443 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:19:54.443 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:19:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:19:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:19:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:19:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:19:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:19:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:19:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2022-04-10T01:19:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:19:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:19:54.490 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=6 2022-04-10T01:19:54.490 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 10 = 6 2022-04-10T01:19:54.491 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:19:54.491 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:19:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:19:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:19:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:19:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:19:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:19:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:19:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:19:54.894 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:19:54.894 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:19:54.894 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2022-04-10T01:19:54.894 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:19:54.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:19:54.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:19:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:19:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:19:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:19:54.942 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:19:54.942 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 10 = '' 2022-04-10T01:19:54.942 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:19:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:19:55.743 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:19:55.744 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:19:55.744 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:19:55.745 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:19:55.745 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:19:55.745 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:19:55.745 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:19:55.746 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:19:55.746 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:19:55.746 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2022-04-10T01:19:55.747 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:19:55.790 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:19:55.791 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:19:55.792 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:19:55.792 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:19:55.792 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:19:55.793 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:19:55.793 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 10 = '' 2022-04-10T01:19:55.794 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:19:55.794 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1.6 2022-04-10T01:19:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:19:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:19:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:19:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:19:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:19:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:19:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:19:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:19:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:19:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2022-04-10T01:19:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:19:57.434 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:19:57.437 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=10 2022-04-10T01:19:57.437 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 10 = 10 2022-04-10T01:19:57.437 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:19:57.437 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror test snap2 2022-04-10T01:19:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:19:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:19:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:19:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap2 2022-04-10T01:19:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:19:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:19:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:19:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:19:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:19:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:19:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:19:58.437 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/test@snap2 2022-04-10T01:19:58.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'test': 2022-04-10T01:19:58.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 512 MiB in 128 objects 2022-04-10T01:19:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:19:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 5 2022-04-10T01:19:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 10ae4a893004 2022-04-10T01:19:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.10ae4a893004 2022-04-10T01:19:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:19:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, operations, non-primary 2022-04-10T01:19:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: snap-trash 2022-04-10T01:19:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:19:58.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:18:11 2022 2022-04-10T01:19:58.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:18:11 2022 2022-04-10T01:19:58.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:18:11 2022 2022-04-10T01:19:58.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:19:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:19:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:19:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 5ff848f4-a6a0-498d-b9f1-7a041de7de98 2022-04-10T01:19:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:19:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:19:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snap1-clean ']' 2022-04-10T01:19:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror test snap1-clean 2022-04-10T01:19:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:19:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:19:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap1-clean 2022-04-10T01:19:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:19:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:19:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:19:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:19:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:19:59.064 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:19:59.068 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:19:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:19:59.873 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:20:00.124 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:20:00.270 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:20:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror test snap2 2022-04-10T01:20:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:20:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:20:00.272 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap2 2022-04-10T01:20:00.272 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:20:00.272 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:20:00.272 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:20:00.272 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:20:00.273 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:20:00.850 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:20:00.854 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:20:01.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:20:01.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:20:01.898 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:20:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:20:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ clean_snap_name=snap2-clean 2022-04-10T01:20:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror test snap2-clean 2022-04-10T01:20:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:20:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:20:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:20:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap2-clean 2022-04-10T01:20:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2-clean --debug-rbd=20 --debug-journaler=20 2022-04-10T01:20:02.195 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 10` 2022-04-10T01:20:02.195 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror test 2022-04-10T01:20:02.196 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:20:02.196 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:20:02.196 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:20:02.196 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:20:02.198 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=7 2022-04-10T01:20:02.198 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:20:02.198 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:20:02.199 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:20:02.199 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:20:02.199 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:20:02.200 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2022-04-10T01:20:02.428 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 50990080 bytes 2022-04-10T01:20:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 130438144 bytes 2022-04-10T01:20:02.765 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 197456896 bytes 2022-04-10T01:20:02.851 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 209330176 bytes 2022-04-10T01:20:02.933 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 215852032 bytes 2022-04-10T01:20:03.070 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 268452864 bytes 2022-04-10T01:20:03.151 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 276044800 bytes 2022-04-10T01:20:03.245 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 296015872 bytes 2022-04-10T01:20:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 342576128 bytes 2022-04-10T01:20:03.468 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 367926272 bytes 2022-04-10T01:20:03.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 382358528 bytes 2022-04-10T01:20:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 413360128 bytes 2022-04-10T01:20:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 446675968 bytes 2022-04-10T01:20:03.884 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 485786624 bytes 2022-04-10T01:20:03.989 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 511684608 bytes 2022-04-10T01:20:04.096 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 539251712 bytes 2022-04-10T01:20:04.217 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 585298944 bytes 2022-04-10T01:20:04.348 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 614436864 bytes 2022-04-10T01:20:04.420 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 616705024 bytes 2022-04-10T01:20:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 692124672 bytes 2022-04-10T01:20:04.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 745632768 bytes 2022-04-10T01:20:04.827 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 775916544 bytes 2022-04-10T01:20:04.919 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 791791616 bytes 2022-04-10T01:20:05.098 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 878779392 bytes 2022-04-10T01:20:05.197 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 902995968 bytes 2022-04-10T01:20:05.336 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 967962624 bytes 2022-04-10T01:20:05.459 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1003995136 bytes 2022-04-10T01:20:05.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1025036288 bytes 2022-04-10T01:20:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1140986880 bytes 2022-04-10T01:20:05.871 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1154947072 bytes 2022-04-10T01:20:05.978 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1182961664 bytes 2022-04-10T01:20:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1207455744 bytes 2022-04-10T01:20:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1256031232 bytes 2022-04-10T01:20:06.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1338272768 bytes 2022-04-10T01:20:06.541 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1420769280 bytes 2022-04-10T01:20:06.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1456740352 bytes 2022-04-10T01:20:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1520908288 bytes 2022-04-10T01:20:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1593152512 bytes 2022-04-10T01:20:07.129 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1652160512 bytes 2022-04-10T01:20:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1652788224 bytes 2022-04-10T01:20:07.331 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1702344704 bytes 2022-04-10T01:20:07.405 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 1702960128 bytes 2022-04-10T01:20:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 1715493888 bytes 2022-04-10T01:20:07.584 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 1729264640 bytes 2022-04-10T01:20:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 1770157056 bytes 2022-04-10T01:20:07.841 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 1826371584 bytes 2022-04-10T01:20:08.001 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 1899187200 bytes 2022-04-10T01:20:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 1917413376 bytes 2022-04-10T01:20:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 1944541184 bytes 2022-04-10T01:20:08.355 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 2008761344 bytes 2022-04-10T01:20:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 2044918784 bytes 2022-04-10T01:20:08.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 2070974464 bytes 2022-04-10T01:20:08.730 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 2121902080 bytes 2022-04-10T01:20:08.761 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5514607811742779) > weight (0.25) 2022-04-10T01:20:08.761 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.4450311869624872) > weight (0.25) 2022-04-10T01:20:08.762 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.7196404625132846) > weight (0.25) 2022-04-10T01:20:08.762 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2022-04-10T01:20:08.763 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2022-04-10T01:20:08.764 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2022-04-10T01:20:08.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi043.stderr:2022-04-10T01:20:08.763+0000 7f308dc8d700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 39751) UID: 1000 2022-04-10T01:20:08.889 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5400 IOs, 2189534208 bytes 2022-04-10T01:20:08.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi043.stderr:daemon-helper: command failed with exit status 1 2022-04-10T01:20:09.038 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5500 IOs, 2245889024 bytes 2022-04-10T01:20:09.149 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5600 IOs, 2279265280 bytes 2022-04-10T01:20:09.202 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:20:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:20:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:20:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:20:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2022-04-10T01:20:09.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:20:09.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:20:09.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:20:09.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state_pattern=up+replaying 2022-04-10T01:20:09.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local description_pattern= 2022-04-10T01:20:09.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local service_pattern= 2022-04-10T01:20:09.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:20:09.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:20:09.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:20:09.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:20:09.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:20:09.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:20:10.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2022-04-10T01:20:10.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:20:10.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:20:10.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:20:10.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state_pattern=up+replaying 2022-04-10T01:20:10.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local description_pattern= 2022-04-10T01:20:10.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local service_pattern= 2022-04-10T01:20:10.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:20:10.207 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:20:10.207 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:20:10.207 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:20:10.207 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-test.mirror_status 2022-04-10T01:20:10.208 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-test.mirror_status 2022-04-10T01:20:10.208 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:20:10.209 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:20:10.209 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ CEPH_ARGS= 2022-04-10T01:20:10.209 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2022-04-10T01:20:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ tee /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:20:10.252 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:test: 2022-04-10T01:20:10.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: global_id: 5ff848f4-a6a0-498d-b9f1-7a041de7de98 2022-04-10T01:20:10.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: state: up+replaying 2022-04-10T01:20:10.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: description: replaying, {"bytes_per_second":17833267.2,"bytes_per_snapshot":535740928.0,"local_snapshot_timestamp":1649553594,"remote_snapshot_timestamp":1649553594,"replay_state":"idle"} 2022-04-10T01:20:10.254 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: service: mirror.0 on smithi043 2022-04-10T01:20:10.254 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: last_update: 2022-04-10 01:20:09 2022-04-10T01:20:10.254 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: peer_sites: 2022-04-10T01:20:10.254 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: name: cluster2 2022-04-10T01:20:10.255 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: state: up+stopped 2022-04-10T01:20:10.255 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: description: local image is primary 2022-04-10T01:20:10.256 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: last_update: 2022-04-10 01:20:09 2022-04-10T01:20:10.257 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:20:10.259 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: state: up+replaying 2022-04-10T01:20:10.260 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep '^ description: .*' /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:20:10.261 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: description: replaying, {"bytes_per_second":17833267.2,"bytes_per_snapshot":535740928.0,"local_snapshot_timestamp":1649553594,"remote_snapshot_timestamp":1649553594,"replay_state":"idle"} 2022-04-10T01:20:10.262 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:20:10.262 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo up+replaying 2022-04-10T01:20:10.263 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep '^up+' 2022-04-10T01:20:10.264 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:up+replaying 2022-04-10T01:20:10.264 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:20:10.265 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: service: mirror.0 on smithi043 2022-04-10T01:20:10.268 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:20:10.268 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local 'last_update=2022-04-10 01:20:09' 2022-04-10T01:20:10.268 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2022-04-10 01:20:09' 2022-04-10T01:20:10.268 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:20:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:20:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:20:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state_pattern=up+replaying 2022-04-10T01:20:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local 'prev_last_update=2022-04-10 01:20:09' 2022-04-10T01:20:10.270 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:20:10.270 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:20:10.270 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:20:10.270 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:20:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror.mirror_status 2022-04-10T01:20:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror.mirror_status 2022-04-10T01:20:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:20:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror.mirror_status 2022-04-10T01:20:10.272 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2022-04-10T01:20:10.300 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: failed to query services: (13) Permission denied 2022-04-10T01:20:10.321 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local last_update state 2022-04-10T01:20:10.322 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2022-04-10T01:20:10.324 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ last_update='2022-04-10 01:20:09' 2022-04-10T01:20:10.324 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2022-04-10T01:20:10.326 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ state=up+replaying 2022-04-10T01:20:10.326 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo up+replaying 2022-04-10T01:20:10.327 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep up+replaying 2022-04-10T01:20:10.327 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:up+replaying 2022-04-10T01:20:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:20:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:20:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ snap_name=snap3 2022-04-10T01:20:10.330 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror test snap3 2022-04-10T01:20:10.330 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:20:10.330 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:20:10.330 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:20:10.330 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap3 2022-04-10T01:20:10.331 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:20:12.769 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7695996617584691) > weight (0.25) 2022-04-10T01:20:12.770 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.49328771093836943) > weight (0.25) 2022-04-10T01:20:12.770 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2022-04-10T01:20:12.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2022-04-10T01:20:12.771 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2022-04-10T01:20:12.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi043.stderr:2022-04-10T01:20:12.770+0000 7ff9040e8700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 39513) UID: 1000 2022-04-10T01:20:12.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi043.stderr:daemon-helper: command failed with exit status 1 2022-04-10T01:20:17.765 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2022-04-10T01:20:17.765 INFO:teuthology.orchestra.run:waiting for 600 2022-04-10T01:20:17.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-04-10T01:20:17.766 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2022-04-10T01:20:17.766 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2022-04-10T01:20:17.767 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2022-04-10T01:20:17.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2022-04-10T01:20:17.771 INFO:tasks.rbd_mirror_thrash:waiting for 67 secs before thrashing 2022-04-10T01:20:18.773 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2022-04-10T01:20:18.773 INFO:teuthology.orchestra.run:waiting for 600 2022-04-10T01:20:18.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-04-10T01:20:18.774 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2022-04-10T01:20:18.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2022-04-10T01:20:18.774 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2022-04-10T01:20:18.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2022-04-10T01:20:18.778 INFO:tasks.rbd_mirror_thrash:waiting for 66 secs before thrashing 2022-04-10T01:20:40.201 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror test 2022-04-10T01:20:40.202 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:20:40.202 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:20:40.202 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:20:40.202 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:20:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:20:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:20:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:20:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2022-04-10T01:20:40.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:20:40.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:20:40.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:20:40.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:20:40.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:20:40.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:20:40.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:20:40.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:20:40.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:20:40.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:20:40.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:20:41.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror test started 2022-04-10T01:20:41.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:20:41.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:20:41.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:20:41.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:20:41.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:20:41.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:20:41.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:20:41.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:20:41.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:20:41.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:20:41.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2022-04-10T01:20:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:20:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:20:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:20:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:20:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:20:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:20:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:20:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:20:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:20:41.208 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:20:41.208 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:20:41.208 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:20:41.212 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:20:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:20:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:20:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2022-04-10T01:20:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:20:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:20:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:20:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:20:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:20:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:20:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:20:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:20:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:20:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:20:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:20:41.216 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:20:41.216 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:20:41.216 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:20:41.216 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:20:41.216 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:20:41.217 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:20:41.217 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:20:41.217 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:20:41.217 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:20:41.217 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:20:41.218 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:20:41.218 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:20:41.218 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:20:41.218 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:20:41.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:20:41.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:20:41.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:20:41.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:20:41.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:20:41.220 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:20:41.220 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:20:41.220 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:20:41.220 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:20:41.220 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:20:41.221 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:20:41.221 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:20:41.221 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:20:41.225 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:20:41.226 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:20:41.226 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:20:41.226 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2022-04-10T01:20:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:20:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:20:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:20:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:20:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:20:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:20:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:20:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:20:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2022-04-10T01:20:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:20:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:20:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:20:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:20:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:20:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:20:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:20:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:20:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:20:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:20:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2022-04-10T01:20:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:20:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:20:41.350 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:20:41.350 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:20:41.350 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:20:41.350 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:20:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:20:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:20:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:20:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:20:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:20:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:20:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-test.status 2022-04-10T01:20:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-test.status 2022-04-10T01:20:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:20:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.status 2022-04-10T01:20:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-test.status 2022-04-10T01:20:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-test.status 2022-04-10T01:20:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:20:41.354 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:20:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror test 2022-04-10T01:20:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:20:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:20:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:20:41.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:20:41.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:20:41.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:20:41.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:20:41.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2022-04-10T01:20:42.208 INFO:tasks.ceph.cluster2.osd.1.smithi043.stderr:2022-04-10T01:20:42.206+0000 7f18011ce700 -1 reset not still connected to 0x55b2c5f64750 2022-04-10T01:20:42.209 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 14 2022-04-10T01:20:42.218 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.rgwz74WspX/cluster2-mirror-test.status 2022-04-10T01:20:42.218 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:20:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:20:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:20:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.status 2022-04-10T01:20:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:20:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:20:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:20:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:20:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2022-04-10T01:20:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:20:42.263 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:20:42.265 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=14 2022-04-10T01:20:42.265 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:20:42.266 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:20:42.266 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:20:42.467 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:20:42.467 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:20:42.468 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:20:42.468 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:20:42.468 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:20:42.468 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:20:42.469 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:20:42.469 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:20:42.469 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:20:42.469 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2022-04-10T01:20:42.469 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:20:42.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:20:42.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=10 2022-04-10T01:20:42.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 14 = 10 2022-04-10T01:20:42.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:20:42.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:20:42.918 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:20:42.919 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:20:42.919 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:20:42.919 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:20:42.919 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:20:42.920 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:20:42.920 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:20:42.920 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:20:42.920 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:20:42.920 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2022-04-10T01:20:42.921 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:20:42.966 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:20:42.967 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:20:42.968 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:20:42.969 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:20:42.969 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:20:42.969 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:20:42.969 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 14 = '' 2022-04-10T01:20:42.970 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:20:42.970 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:20:43.772 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:20:43.773 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:20:43.773 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:20:43.773 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:20:43.774 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:20:43.774 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:20:43.774 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:20:43.774 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:20:43.775 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:20:43.775 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2022-04-10T01:20:43.775 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:20:43.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:20:43.819 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:20:43.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:20:43.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:20:43.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:20:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:20:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 14 = '' 2022-04-10T01:20:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:20:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1.6 2022-04-10T01:20:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:20:45.423 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:20:45.423 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:20:45.424 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:20:45.424 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:20:45.424 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:20:45.424 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:20:45.425 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:20:45.425 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:20:45.426 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2022-04-10T01:20:45.426 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:20:45.464 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:20:45.466 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=14 2022-04-10T01:20:45.466 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 14 = 14 2022-04-10T01:20:45.466 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:20:45.466 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror test snap3 2022-04-10T01:20:45.467 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:20:45.467 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:20:45.467 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:20:45.467 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap3 2022-04-10T01:20:45.468 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:20:45.468 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:20:45.468 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:20:45.468 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:20:45.468 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:20:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:20:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:20:46.466 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/test@snap3 2022-04-10T01:20:46.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'test': 2022-04-10T01:20:46.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 512 MiB in 128 objects 2022-04-10T01:20:46.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:20:46.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 7 2022-04-10T01:20:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 10ae4a893004 2022-04-10T01:20:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.10ae4a893004 2022-04-10T01:20:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:20:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, operations, non-primary 2022-04-10T01:20:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: snap-trash 2022-04-10T01:20:46.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:20:46.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:18:11 2022 2022-04-10T01:20:46.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:19:59 2022 2022-04-10T01:20:46.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:18:11 2022 2022-04-10T01:20:46.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:20:46.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:20:46.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:20:46.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 5ff848f4-a6a0-498d-b9f1-7a041de7de98 2022-04-10T01:20:46.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:20:46.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:20:46.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snap2-clean ']' 2022-04-10T01:20:46.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror test snap2-clean 2022-04-10T01:20:46.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:20:46.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:20:46.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap2-clean 2022-04-10T01:20:46.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:20:46.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:20:46.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:20:46.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:20:46.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:20:47.100 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:20:47.104 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:20:47.879 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:20:47.882 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:20:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:20:48.281 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:20:48.281 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror test snap3 2022-04-10T01:20:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:20:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:20:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap3 2022-04-10T01:20:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:20:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:20:48.283 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:20:48.283 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:20:48.283 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:20:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:20:48.873 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:20:49.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:20:49.777 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:20:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:20:50.023 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:20:50.023 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ clean_snap_name=snap3-clean 2022-04-10T01:20:50.023 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror test snap3-clean 2022-04-10T01:20:50.023 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:20:50.024 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:20:50.024 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:20:50.024 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap3-clean 2022-04-10T01:20:50.024 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3-clean --debug-rbd=20 --debug-journaler=20 2022-04-10T01:20:50.170 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 10` 2022-04-10T01:20:50.170 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror test 2022-04-10T01:20:50.171 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:20:50.171 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:20:50.171 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:20:50.171 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:20:50.173 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=6 2022-04-10T01:20:50.173 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:20:50.173 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:20:50.173 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:20:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:20:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:20:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2022-04-10T01:20:50.358 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 613376 bytes 2022-04-10T01:20:50.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 30243840 bytes 2022-04-10T01:20:50.629 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 70969344 bytes 2022-04-10T01:20:50.742 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 99098624 bytes 2022-04-10T01:20:50.833 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 110404608 bytes 2022-04-10T01:20:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 131576832 bytes 2022-04-10T01:20:51.048 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 159661056 bytes 2022-04-10T01:20:51.163 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 180591616 bytes 2022-04-10T01:20:51.276 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 208455680 bytes 2022-04-10T01:20:51.414 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 254511104 bytes 2022-04-10T01:20:51.489 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 255140864 bytes 2022-04-10T01:20:51.600 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 293572608 bytes 2022-04-10T01:20:51.771 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 366808064 bytes 2022-04-10T01:20:51.943 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 444999680 bytes 2022-04-10T01:20:52.105 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 514874368 bytes 2022-04-10T01:20:52.230 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 557492224 bytes 2022-04-10T01:20:52.389 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 617282560 bytes 2022-04-10T01:20:52.588 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 719747072 bytes 2022-04-10T01:20:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 743511040 bytes 2022-04-10T01:20:52.826 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 796303360 bytes 2022-04-10T01:20:52.951 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 827710464 bytes 2022-04-10T01:20:53.081 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 872314880 bytes 2022-04-10T01:20:53.307 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 988630016 bytes 2022-04-10T01:20:53.412 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 1007673344 bytes 2022-04-10T01:20:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 1014548480 bytes 2022-04-10T01:20:53.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1085204480 bytes 2022-04-10T01:20:53.797 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1135206400 bytes 2022-04-10T01:20:53.987 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1233498112 bytes 2022-04-10T01:20:54.101 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1256890368 bytes 2022-04-10T01:20:54.233 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1287380992 bytes 2022-04-10T01:20:54.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1323580416 bytes 2022-04-10T01:20:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1380214784 bytes 2022-04-10T01:20:54.630 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1425807360 bytes 2022-04-10T01:20:54.769 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1470489600 bytes 2022-04-10T01:20:54.928 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1549572096 bytes 2022-04-10T01:20:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1555759104 bytes 2022-04-10T01:20:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1587446784 bytes 2022-04-10T01:20:55.298 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1634072576 bytes 2022-04-10T01:20:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1653357568 bytes 2022-04-10T01:20:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1718862848 bytes 2022-04-10T01:20:55.722 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1782031360 bytes 2022-04-10T01:20:55.845 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 1811603456 bytes 2022-04-10T01:20:55.972 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 1847738368 bytes 2022-04-10T01:20:56.089 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 1878640640 bytes 2022-04-10T01:20:56.179 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:20:56.179 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:20:56.180 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:20:56.180 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:20:56.180 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2022-04-10T01:20:56.180 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:20:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:20:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:20:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state_pattern=up+replaying 2022-04-10T01:20:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local description_pattern= 2022-04-10T01:20:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local service_pattern= 2022-04-10T01:20:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:20:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:20:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:20:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:20:56.183 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:20:56.183 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:20:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2022-04-10T01:20:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:20:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:20:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:20:57.182 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state_pattern=up+replaying 2022-04-10T01:20:57.182 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local description_pattern= 2022-04-10T01:20:57.182 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local service_pattern= 2022-04-10T01:20:57.183 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:20:57.183 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:20:57.183 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:20:57.183 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:20:57.184 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-test.mirror_status 2022-04-10T01:20:57.184 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-test.mirror_status 2022-04-10T01:20:57.184 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:20:57.185 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:20:57.185 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ CEPH_ARGS= 2022-04-10T01:20:57.185 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2022-04-10T01:20:57.186 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ tee /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:20:57.231 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:test: 2022-04-10T01:20:57.231 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: global_id: 5ff848f4-a6a0-498d-b9f1-7a041de7de98 2022-04-10T01:20:57.231 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: state: up+replaying 2022-04-10T01:20:57.231 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":535740928.0,"local_snapshot_timestamp":1649553594,"remote_snapshot_timestamp":1649553594,"replay_state":"idle"} 2022-04-10T01:20:57.232 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: service: mirror.0 on smithi043 2022-04-10T01:20:57.232 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: last_update: 2022-04-10 01:20:39 2022-04-10T01:20:57.232 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: peer_sites: 2022-04-10T01:20:57.232 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: name: cluster2 2022-04-10T01:20:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: state: up+stopped 2022-04-10T01:20:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: description: local image is primary 2022-04-10T01:20:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: last_update: 2022-04-10 01:20:39 2022-04-10T01:20:57.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:20:57.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: state: up+replaying 2022-04-10T01:20:57.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep '^ description: .*' /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:20:57.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":535740928.0,"local_snapshot_timestamp":1649553594,"remote_snapshot_timestamp":1649553594,"replay_state":"idle"} 2022-04-10T01:20:57.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:20:57.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo up+replaying 2022-04-10T01:20:57.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep '^up+' 2022-04-10T01:20:57.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:up+replaying 2022-04-10T01:20:57.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:20:57.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: service: mirror.0 on smithi043 2022-04-10T01:20:57.242 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:20:57.244 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local 'last_update=2022-04-10 01:20:39' 2022-04-10T01:20:57.244 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2022-04-10 01:20:39' 2022-04-10T01:20:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:20:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:20:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:20:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state_pattern=up+replaying 2022-04-10T01:20:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local 'prev_last_update=2022-04-10 01:20:39' 2022-04-10T01:20:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:20:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:20:57.247 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:20:57.247 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:20:57.247 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror.mirror_status 2022-04-10T01:20:57.247 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror.mirror_status 2022-04-10T01:20:57.247 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:20:57.248 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror.mirror_status 2022-04-10T01:20:57.248 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2022-04-10T01:20:57.280 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: failed to query services: (13) Permission denied 2022-04-10T01:20:57.302 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local last_update state 2022-04-10T01:20:57.303 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2022-04-10T01:20:57.304 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ last_update='2022-04-10 01:20:39' 2022-04-10T01:20:57.305 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2022-04-10T01:20:57.306 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ state=up+replaying 2022-04-10T01:20:57.307 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo up+replaying 2022-04-10T01:20:57.307 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep up+replaying 2022-04-10T01:20:57.308 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:up+replaying 2022-04-10T01:20:57.309 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:20:57.309 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:20:57.310 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ snap_name=snap4 2022-04-10T01:20:57.310 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror test snap4 2022-04-10T01:20:57.310 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:20:57.310 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:20:57.310 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:20:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap4 2022-04-10T01:20:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:21:24.772 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5282912996431222) > weight (0.25) 2022-04-10T01:21:24.773 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.3372490953425642) > weight (0.25) 2022-04-10T01:21:24.773 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2022-04-10T01:21:24.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2022-04-10T01:21:24.774 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2022-04-10T01:21:24.775 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi043.stderr:2022-04-10T01:21:24.774+0000 7f484bb29700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 39685) UID: 1000 2022-04-10T01:21:24.779 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6561301251771614) > weight (0.25) 2022-04-10T01:21:24.779 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.807607315642758) > weight (0.25) 2022-04-10T01:21:24.779 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.5915256965522383) > weight (0.25) 2022-04-10T01:21:24.779 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.31248070423073304) > weight (0.25) 2022-04-10T01:21:24.780 INFO:tasks.rbd_mirror_thrash:waiting for 64 secs before thrashing 2022-04-10T01:21:24.975 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi043.stderr:daemon-helper: command failed with exit status 1 2022-04-10T01:21:27.218 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror test 2022-04-10T01:21:27.218 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:21:27.218 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:21:27.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:21:27.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:21:27.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:21:27.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:21:27.220 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:21:27.220 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2022-04-10T01:21:27.220 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:21:27.220 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:21:27.221 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:21:27.221 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:21:27.221 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:21:27.221 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:21:27.222 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:21:27.222 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:21:27.222 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:21:27.222 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:21:27.222 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:21:28.220 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror test started 2022-04-10T01:21:28.220 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:21:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:21:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:21:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:21:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:21:28.222 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:21:28.222 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:21:28.222 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:21:28.222 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:21:28.223 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:21:28.223 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2022-04-10T01:21:28.223 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:21:28.223 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:21:28.224 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:21:28.224 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:21:28.224 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:21:28.225 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:21:28.225 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:21:28.225 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:21:28.225 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:21:28.226 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:21:28.226 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:21:28.226 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:21:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:21:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:21:28.232 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:21:28.232 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2022-04-10T01:21:28.232 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:21:28.232 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:21:28.233 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:21:28.233 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:21:28.233 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:21:28.234 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:21:28.234 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:21:28.234 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:21:28.235 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:21:28.235 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:21:28.235 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:21:28.235 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:21:28.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:21:28.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:21:28.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:21:28.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:21:28.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:21:28.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:21:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:21:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:21:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:21:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:21:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:21:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:21:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:21:28.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:21:28.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:21:28.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:21:28.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:21:28.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:21:28.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:21:28.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:21:28.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:21:28.242 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:21:28.242 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:21:28.242 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:21:28.243 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:21:28.243 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:21:28.244 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:21:28.244 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:21:28.244 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:21:28.245 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2022-04-10T01:21:28.357 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:21:28.358 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:21:28.358 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:21:28.359 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:21:28.359 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:21:28.363 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:21:28.364 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:21:28.364 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:21:28.364 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2022-04-10T01:21:28.365 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:21:28.365 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:21:28.365 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:21:28.365 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:21:28.366 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:21:28.366 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:21:28.366 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:21:28.366 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:21:28.366 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:21:28.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:21:28.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2022-04-10T01:21:28.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:21:28.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:21:28.368 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:21:28.368 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:21:28.368 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:21:28.368 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:21:28.368 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:21:28.369 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:21:28.369 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:21:28.369 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:21:28.369 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:21:28.369 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:21:28.370 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-test.status 2022-04-10T01:21:28.370 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-test.status 2022-04-10T01:21:28.370 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:21:28.370 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.status 2022-04-10T01:21:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-test.status 2022-04-10T01:21:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-test.status 2022-04-10T01:21:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:21:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:21:28.372 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror test 2022-04-10T01:21:28.372 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:21:28.372 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:21:28.372 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:21:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:21:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:21:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:21:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:21:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2022-04-10T01:21:29.221 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 18 2022-04-10T01:21:29.233 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.rgwz74WspX/cluster2-mirror-test.status 2022-04-10T01:21:29.234 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:21:29.234 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:21:29.234 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:21:29.235 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.status 2022-04-10T01:21:29.235 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:21:29.235 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:21:29.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:21:29.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:21:29.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2022-04-10T01:21:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:21:29.278 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:21:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=18 2022-04-10T01:21:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:21:29.282 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:21:29.282 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:21:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:21:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:21:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:21:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:21:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:21:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:21:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:21:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:21:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:21:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2022-04-10T01:21:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:21:29.528 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:21:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=14 2022-04-10T01:21:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 18 = 14 2022-04-10T01:21:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:21:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:21:29.775 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2022-04-10T01:21:29.776 INFO:teuthology.orchestra.run:waiting for 600 2022-04-10T01:21:29.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-04-10T01:21:29.777 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2022-04-10T01:21:29.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2022-04-10T01:21:29.778 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2022-04-10T01:21:29.782 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2022-04-10T01:21:29.782 INFO:tasks.rbd_mirror_thrash:waiting for 91 secs before thrashing 2022-04-10T01:21:29.932 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:21:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:21:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:21:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:21:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:21:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:21:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:21:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:21:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:21:29.935 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2022-04-10T01:21:29.935 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:21:29.986 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:21:29.987 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:21:29.988 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:21:29.988 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:21:29.988 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:21:29.989 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:21:29.990 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 18 = '' 2022-04-10T01:21:29.990 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:21:29.990 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:21:30.791 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:21:30.791 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:21:30.792 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:21:30.792 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:21:30.792 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:21:30.792 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:21:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:21:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:21:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:21:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2022-04-10T01:21:30.794 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:21:30.840 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:21:30.841 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:21:30.842 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:21:30.842 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:21:30.842 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:21:30.843 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:21:30.843 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 18 = '' 2022-04-10T01:21:30.843 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:21:30.844 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1.6 2022-04-10T01:21:32.444 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:21:32.445 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:21:32.445 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:21:32.445 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:21:32.446 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:21:32.446 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:21:32.446 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:21:32.446 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:21:32.446 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:21:32.447 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2022-04-10T01:21:32.447 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:21:32.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:21:32.490 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=18 2022-04-10T01:21:32.491 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 18 = 18 2022-04-10T01:21:32.491 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:21:32.491 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror test snap4 2022-04-10T01:21:32.491 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:21:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:21:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:21:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap4 2022-04-10T01:21:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:21:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:21:32.493 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:21:32.493 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:21:32.493 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:21:32.493 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:21:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:21:33.491 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/test@snap4 2022-04-10T01:21:33.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'test': 2022-04-10T01:21:33.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 512 MiB in 128 objects 2022-04-10T01:21:33.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:21:33.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 9 2022-04-10T01:21:33.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 10ae4a893004 2022-04-10T01:21:33.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.10ae4a893004 2022-04-10T01:21:33.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:21:33.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, operations, non-primary 2022-04-10T01:21:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: snap-trash 2022-04-10T01:21:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:21:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:18:11 2022 2022-04-10T01:21:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:19:59 2022 2022-04-10T01:21:33.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:18:11 2022 2022-04-10T01:21:33.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:21:33.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:21:33.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:21:33.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 5ff848f4-a6a0-498d-b9f1-7a041de7de98 2022-04-10T01:21:33.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:21:33.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:21:33.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snap3-clean ']' 2022-04-10T01:21:33.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror test snap3-clean 2022-04-10T01:21:33.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:21:33.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:21:33.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap3-clean 2022-04-10T01:21:33.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:21:33.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:21:33.541 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:21:33.541 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:21:33.541 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:21:34.304 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:21:34.307 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:21:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:21:35.103 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:21:35.358 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:21:35.502 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:21:35.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror test snap4 2022-04-10T01:21:35.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:21:35.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:21:35.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap4 2022-04-10T01:21:35.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:21:35.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:21:35.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:21:35.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:21:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:21:36.294 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:21:36.298 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:21:37.075 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:21:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:21:37.335 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:21:37.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:21:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ clean_snap_name=snap4-clean 2022-04-10T01:21:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror test snap4-clean 2022-04-10T01:21:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:21:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:21:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:21:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap4-clean 2022-04-10T01:21:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4-clean --debug-rbd=20 --debug-journaler=20 2022-04-10T01:21:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 10` 2022-04-10T01:21:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror test 2022-04-10T01:21:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:21:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:21:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:21:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:21:38.193 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=11 2022-04-10T01:21:38.193 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:21:38.194 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:21:38.194 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:21:38.194 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:21:38.194 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:21:38.195 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2022-04-10T01:21:38.463 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 55952384 bytes 2022-04-10T01:21:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 133771264 bytes 2022-04-10T01:21:38.781 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 148070400 bytes 2022-04-10T01:21:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 170857472 bytes 2022-04-10T01:21:39.007 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 195464192 bytes 2022-04-10T01:21:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 203900928 bytes 2022-04-10T01:21:39.225 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 228032512 bytes 2022-04-10T01:21:39.391 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 311119872 bytes 2022-04-10T01:21:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 347015168 bytes 2022-04-10T01:21:39.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 369715200 bytes 2022-04-10T01:21:39.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 448022528 bytes 2022-04-10T01:21:40.028 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 523580416 bytes 2022-04-10T01:21:40.128 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 539011072 bytes 2022-04-10T01:21:40.244 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 562896896 bytes 2022-04-10T01:21:40.332 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 563513344 bytes 2022-04-10T01:21:40.447 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 590827520 bytes 2022-04-10T01:21:40.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 676614144 bytes 2022-04-10T01:21:40.750 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 695862272 bytes 2022-04-10T01:21:40.853 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 723111936 bytes 2022-04-10T01:21:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 754121728 bytes 2022-04-10T01:21:41.142 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 816234496 bytes 2022-04-10T01:21:41.260 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 855433216 bytes 2022-04-10T01:21:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 856050688 bytes 2022-04-10T01:21:41.614 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 989622272 bytes 2022-04-10T01:21:41.732 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 1002181632 bytes 2022-04-10T01:21:41.914 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1086152704 bytes 2022-04-10T01:21:42.078 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1151585280 bytes 2022-04-10T01:21:42.208 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1196929024 bytes 2022-04-10T01:21:42.334 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1220370432 bytes 2022-04-10T01:21:42.502 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1306475520 bytes 2022-04-10T01:21:42.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1346301952 bytes 2022-04-10T01:21:42.754 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1349518336 bytes 2022-04-10T01:21:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1350164480 bytes 2022-04-10T01:21:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1375014912 bytes 2022-04-10T01:21:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1398804480 bytes 2022-04-10T01:21:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1430287360 bytes 2022-04-10T01:21:43.390 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1520474112 bytes 2022-04-10T01:21:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1566759936 bytes 2022-04-10T01:21:43.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1595472896 bytes 2022-04-10T01:21:43.774 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1630597120 bytes 2022-04-10T01:21:43.926 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1680507904 bytes 2022-04-10T01:21:44.020 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 1698038784 bytes 2022-04-10T01:21:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 1730573312 bytes 2022-04-10T01:21:44.303 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 1768718336 bytes 2022-04-10T01:21:44.417 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 1790462976 bytes 2022-04-10T01:21:44.583 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 1866130432 bytes 2022-04-10T01:21:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 1926581248 bytes 2022-04-10T01:21:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 1996495872 bytes 2022-04-10T01:21:45.049 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 2050278400 bytes 2022-04-10T01:21:45.316 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 2160723968 bytes 2022-04-10T01:21:45.484 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 2218288128 bytes 2022-04-10T01:21:45.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 2258265088 bytes 2022-04-10T01:21:45.816 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 2321622016 bytes 2022-04-10T01:21:45.997 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5400 IOs, 2365142016 bytes 2022-04-10T01:21:46.122 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5500 IOs, 2396624896 bytes 2022-04-10T01:21:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5600 IOs, 2456582144 bytes 2022-04-10T01:21:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5700 IOs, 2457180160 bytes 2022-04-10T01:21:46.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5800 IOs, 2493481984 bytes 2022-04-10T01:21:46.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5900 IOs, 2517976064 bytes 2022-04-10T01:21:46.731 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6000 IOs, 2518597632 bytes 2022-04-10T01:21:46.859 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6100 IOs, 2555126784 bytes 2022-04-10T01:21:46.971 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6200 IOs, 2572924928 bytes 2022-04-10T01:21:47.093 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6300 IOs, 2602334208 bytes 2022-04-10T01:21:47.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6400 IOs, 2651024384 bytes 2022-04-10T01:21:47.382 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6500 IOs, 2694343680 bytes 2022-04-10T01:21:47.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6600 IOs, 2718042112 bytes 2022-04-10T01:21:47.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6700 IOs, 2754782208 bytes 2022-04-10T01:21:47.797 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6800 IOs, 2800082944 bytes 2022-04-10T01:21:47.937 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6900 IOs, 2837770240 bytes 2022-04-10T01:21:48.090 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7000 IOs, 2889819136 bytes 2022-04-10T01:21:48.246 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7100 IOs, 2937932800 bytes 2022-04-10T01:21:48.387 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7200 IOs, 2979990528 bytes 2022-04-10T01:21:48.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7300 IOs, 3037853696 bytes 2022-04-10T01:21:48.743 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7400 IOs, 3122006016 bytes 2022-04-10T01:21:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7500 IOs, 3157787648 bytes 2022-04-10T01:21:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7600 IOs, 3158397952 bytes 2022-04-10T01:21:49.083 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7700 IOs, 3177572352 bytes 2022-04-10T01:21:49.200 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:21:49.201 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:21:49.201 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:21:49.202 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:21:49.202 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2022-04-10T01:21:49.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:21:49.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:21:49.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:21:49.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state_pattern=up+replaying 2022-04-10T01:21:49.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local description_pattern= 2022-04-10T01:21:49.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local service_pattern= 2022-04-10T01:21:49.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:21:49.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:21:49.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:21:49.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:21:49.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:21:49.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:21:50.201 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2022-04-10T01:21:50.201 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:21:50.202 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:21:50.202 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:21:50.202 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state_pattern=up+replaying 2022-04-10T01:21:50.202 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local description_pattern= 2022-04-10T01:21:50.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local service_pattern= 2022-04-10T01:21:50.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:21:50.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:21:50.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:21:50.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:21:50.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-test.mirror_status 2022-04-10T01:21:50.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-test.mirror_status 2022-04-10T01:21:50.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:21:50.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:21:50.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ CEPH_ARGS= 2022-04-10T01:21:50.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2022-04-10T01:21:50.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ tee /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:21:50.251 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:test: 2022-04-10T01:21:50.252 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: global_id: 5ff848f4-a6a0-498d-b9f1-7a041de7de98 2022-04-10T01:21:50.252 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: state: up+replaying 2022-04-10T01:21:50.252 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: description: replaying, {"bytes_per_second":17422779.73,"bytes_per_snapshot":528059392.0,"local_snapshot_timestamp":1649553689,"remote_snapshot_timestamp":1649553689,"replay_state":"idle"} 2022-04-10T01:21:50.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: service: mirror.0 on smithi043 2022-04-10T01:21:50.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: last_update: 2022-04-10 01:21:39 2022-04-10T01:21:50.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: peer_sites: 2022-04-10T01:21:50.254 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: name: cluster2 2022-04-10T01:21:50.254 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: state: up+stopped 2022-04-10T01:21:50.254 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: description: local image is primary 2022-04-10T01:21:50.254 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: last_update: 2022-04-10 01:21:39 2022-04-10T01:21:50.256 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:21:50.257 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: state: up+replaying 2022-04-10T01:21:50.258 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep '^ description: .*' /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:21:50.258 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: description: replaying, {"bytes_per_second":17422779.73,"bytes_per_snapshot":528059392.0,"local_snapshot_timestamp":1649553689,"remote_snapshot_timestamp":1649553689,"replay_state":"idle"} 2022-04-10T01:21:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:21:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo up+replaying 2022-04-10T01:21:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep '^up+' 2022-04-10T01:21:50.260 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:up+replaying 2022-04-10T01:21:50.261 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:21:50.261 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: service: mirror.0 on smithi043 2022-04-10T01:21:50.262 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:21:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local 'last_update=2022-04-10 01:21:39' 2022-04-10T01:21:50.264 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2022-04-10 01:21:39' 2022-04-10T01:21:50.264 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:21:50.264 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:21:50.264 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:21:50.265 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state_pattern=up+replaying 2022-04-10T01:21:50.265 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local 'prev_last_update=2022-04-10 01:21:39' 2022-04-10T01:21:50.265 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:21:50.265 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:21:50.266 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:21:50.266 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:21:50.266 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror.mirror_status 2022-04-10T01:21:50.266 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror.mirror_status 2022-04-10T01:21:50.267 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:21:50.267 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror.mirror_status 2022-04-10T01:21:50.267 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2022-04-10T01:21:50.299 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: failed to query services: (13) Permission denied 2022-04-10T01:21:50.323 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local last_update state 2022-04-10T01:21:50.323 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2022-04-10T01:21:50.325 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ last_update='2022-04-10 01:21:39' 2022-04-10T01:21:50.325 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2022-04-10T01:21:50.327 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ state=up+replaying 2022-04-10T01:21:50.327 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo up+replaying 2022-04-10T01:21:50.328 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep up+replaying 2022-04-10T01:21:50.328 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:up+replaying 2022-04-10T01:21:50.329 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:21:50.330 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:21:50.330 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ snap_name=snap5 2022-04-10T01:21:50.330 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror test snap5 2022-04-10T01:21:50.330 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:21:50.331 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:21:50.331 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:21:50.331 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap5 2022-04-10T01:21:50.331 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:22:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror test 2022-04-10T01:22:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:22:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:22:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:22:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:22:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:22:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:22:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:22:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2022-04-10T01:22:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:22:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:22:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:22:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:22:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:22:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:22:20.225 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:22:20.225 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:22:20.225 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:22:20.225 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:22:20.225 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:22:21.222 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror test started 2022-04-10T01:22:21.223 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:22:21.223 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:22:21.223 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:22:21.224 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:22:21.224 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:22:21.224 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:22:21.224 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:22:21.225 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:22:21.225 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:22:21.225 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:22:21.225 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2022-04-10T01:22:21.226 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:22:21.226 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:22:21.226 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:22:21.226 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:22:21.227 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:22:21.227 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:22:21.227 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:22:21.227 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:22:21.227 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:22:21.228 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:22:21.228 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:22:21.228 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:22:21.228 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:22:21.232 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:22:21.233 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:22:21.233 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2022-04-10T01:22:21.233 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:22:21.234 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:22:21.234 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:22:21.234 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:22:21.234 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:22:21.234 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:22:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:22:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:22:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:22:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:22:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:22:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:22:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:22:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:22:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:22:21.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:22:21.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:22:21.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:22:21.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:22:21.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:22:21.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:22:21.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:22:21.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:22:21.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:22:21.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:22:21.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:22:21.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:22:21.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:22:21.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:22:21.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:22:21.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:22:21.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:22:21.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:22:21.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:22:21.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:22:21.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:22:21.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:22:21.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:22:21.244 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:22:21.245 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:22:21.245 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:22:21.245 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2022-04-10T01:22:21.357 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:22:21.357 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:22:21.358 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:22:21.358 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:22:21.359 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:22:21.363 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:22:21.364 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:22:21.364 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:22:21.364 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2022-04-10T01:22:21.364 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:22:21.365 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:22:21.365 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:22:21.365 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:22:21.365 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:22:21.365 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:22:21.366 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:22:21.366 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:22:21.366 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:22:21.366 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:22:21.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2022-04-10T01:22:21.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:22:21.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:22:21.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:22:21.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:22:21.368 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:22:21.368 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:22:21.368 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:22:21.368 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:22:21.369 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:22:21.369 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:22:21.369 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:22:21.369 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:22:21.369 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-test.status 2022-04-10T01:22:21.370 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-test.status 2022-04-10T01:22:21.370 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:22:21.370 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.status 2022-04-10T01:22:21.370 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-test.status 2022-04-10T01:22:21.371 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-test.status 2022-04-10T01:22:21.371 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:22:21.371 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:22:21.371 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror test 2022-04-10T01:22:21.372 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:22:21.372 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:22:21.372 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:22:21.372 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:22:21.373 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:22:21.373 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:22:21.374 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:22:21.374 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2022-04-10T01:22:22.225 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 22 2022-04-10T01:22:22.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.rgwz74WspX/cluster2-mirror-test.status 2022-04-10T01:22:22.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:22:22.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:22:22.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:22:22.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.status 2022-04-10T01:22:22.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:22:22.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:22:22.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:22:22.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:22:22.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2022-04-10T01:22:22.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:22:22.283 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:22:22.285 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=22 2022-04-10T01:22:22.285 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:22:22.286 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:22:22.286 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:22:22.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:22:22.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:22:22.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:22:22.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:22:22.489 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:22:22.489 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:22:22.489 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:22:22.489 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:22:22.490 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:22:22.490 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2022-04-10T01:22:22.490 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:22:22.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:22:22.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=18 2022-04-10T01:22:22.541 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 22 = 18 2022-04-10T01:22:22.541 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:22:22.541 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:22:22.942 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:22:22.943 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:22:22.943 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:22:22.943 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:22:22.943 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:22:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:22:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:22:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:22:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:22:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2022-04-10T01:22:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:22:22.991 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:22:22.991 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:22:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:22:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:22:22.993 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:22:22.993 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:22:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 22 = '' 2022-04-10T01:22:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:22:22.995 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:22:23.795 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:22:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:22:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:22:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:22:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:22:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:22:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:22:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:22:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:22:23.798 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2022-04-10T01:22:23.798 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:22:23.844 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:22:23.844 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:22:23.846 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:22:23.846 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:22:23.846 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:22:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:22:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 22 = '' 2022-04-10T01:22:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:22:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1.6 2022-04-10T01:22:25.448 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:22:25.449 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:22:25.449 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:22:25.449 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:22:25.449 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:22:25.450 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:22:25.450 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:22:25.450 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:22:25.450 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:22:25.451 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2022-04-10T01:22:25.451 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:22:25.490 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:22:25.493 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=22 2022-04-10T01:22:25.493 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 22 = 22 2022-04-10T01:22:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:22:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror test snap5 2022-04-10T01:22:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:22:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:22:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:22:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap5 2022-04-10T01:22:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:22:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:22:25.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:22:25.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:22:25.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:22:25.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:22:25.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:22:26.493 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/test@snap5 2022-04-10T01:22:26.733 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'test': 2022-04-10T01:22:26.733 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 512 MiB in 128 objects 2022-04-10T01:22:26.733 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:22:26.734 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 10 2022-04-10T01:22:26.734 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 10ae4a893004 2022-04-10T01:22:26.734 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.10ae4a893004 2022-04-10T01:22:26.734 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:22:26.734 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:22:26.735 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:22:26.735 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:22:26.735 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:18:11 2022 2022-04-10T01:22:26.735 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:21:34 2022 2022-04-10T01:22:26.736 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:18:11 2022 2022-04-10T01:22:26.736 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:22:26.736 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:22:26.736 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:22:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 5ff848f4-a6a0-498d-b9f1-7a041de7de98 2022-04-10T01:22:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:22:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:22:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snap4-clean ']' 2022-04-10T01:22:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror test snap4-clean 2022-04-10T01:22:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:22:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:22:26.740 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap4-clean 2022-04-10T01:22:26.740 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:22:26.740 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:22:26.740 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:22:26.741 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:22:26.741 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:22:27.312 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:22:27.316 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:22:28.096 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:22:28.100 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:22:28.353 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:22:28.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:22:28.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror test snap5 2022-04-10T01:22:28.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:22:28.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:22:28.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap5 2022-04-10T01:22:28.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:22:28.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:22:28.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:22:28.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:22:28.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:22:28.781 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.25605434073246935) > weight (0.25) 2022-04-10T01:22:28.781 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2022-04-10T01:22:28.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2022-04-10T01:22:28.782 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2022-04-10T01:22:28.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi043.stderr:2022-04-10T01:22:28.782+0000 7fd693b13700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 39470) UID: 1000 2022-04-10T01:22:28.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi043.stderr:daemon-helper: command failed with exit status 1 2022-04-10T01:22:29.079 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:22:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:22:29.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:22:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:22:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:22:30.280 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:22:30.280 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ clean_snap_name=snap5-clean 2022-04-10T01:22:30.280 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror test snap5-clean 2022-04-10T01:22:30.281 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:22:30.281 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:22:30.281 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:22:30.281 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap5-clean 2022-04-10T01:22:30.282 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5-clean --debug-rbd=20 --debug-journaler=20 2022-04-10T01:22:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 10` 2022-04-10T01:22:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror test 2022-04-10T01:22:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:22:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:22:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:22:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:22:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=12 2022-04-10T01:22:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:22:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:22:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:22:31.230 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:22:31.230 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:22:31.230 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2022-04-10T01:22:31.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 48446464 bytes 2022-04-10T01:22:31.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 66688000 bytes 2022-04-10T01:22:31.778 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 90786816 bytes 2022-04-10T01:22:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 110915584 bytes 2022-04-10T01:22:32.028 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 164957184 bytes 2022-04-10T01:22:32.279 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 268069888 bytes 2022-04-10T01:22:32.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 277193728 bytes 2022-04-10T01:22:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 291208192 bytes 2022-04-10T01:22:32.592 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 291823616 bytes 2022-04-10T01:22:32.713 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 313352192 bytes 2022-04-10T01:22:32.783 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2022-04-10T01:22:32.783 INFO:teuthology.orchestra.run:waiting for 600 2022-04-10T01:22:32.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-04-10T01:22:32.784 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2022-04-10T01:22:32.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2022-04-10T01:22:32.785 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2022-04-10T01:22:32.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2022-04-10T01:22:32.789 INFO:tasks.rbd_mirror_thrash:waiting for 111 secs before thrashing 2022-04-10T01:22:32.876 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 362541056 bytes 2022-04-10T01:22:33.077 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 441061376 bytes 2022-04-10T01:22:33.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 491520000 bytes 2022-04-10T01:22:33.462 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 592562176 bytes 2022-04-10T01:22:33.589 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 633756672 bytes 2022-04-10T01:22:33.733 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 656665600 bytes 2022-04-10T01:22:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 669808640 bytes 2022-04-10T01:22:33.952 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 685017088 bytes 2022-04-10T01:22:34.076 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 724057088 bytes 2022-04-10T01:22:34.263 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 774927360 bytes 2022-04-10T01:22:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 808851456 bytes 2022-04-10T01:22:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 843081728 bytes 2022-04-10T01:22:34.704 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 898658304 bytes 2022-04-10T01:22:34.864 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 950022144 bytes 2022-04-10T01:22:35.049 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 1012251648 bytes 2022-04-10T01:22:35.179 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1058583552 bytes 2022-04-10T01:22:35.307 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1067432960 bytes 2022-04-10T01:22:35.422 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1083620352 bytes 2022-04-10T01:22:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1108296704 bytes 2022-04-10T01:22:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1110522880 bytes 2022-04-10T01:22:35.770 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1134717952 bytes 2022-04-10T01:22:35.887 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1155448832 bytes 2022-04-10T01:22:36.024 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1192394752 bytes 2022-04-10T01:22:36.180 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1240678400 bytes 2022-04-10T01:22:36.281 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1241310208 bytes 2022-04-10T01:22:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1284544512 bytes 2022-04-10T01:22:36.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1379433472 bytes 2022-04-10T01:22:36.770 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1420024832 bytes 2022-04-10T01:22:36.921 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1456392192 bytes 2022-04-10T01:22:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1494571008 bytes 2022-04-10T01:22:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1495182336 bytes 2022-04-10T01:22:37.317 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 1549367296 bytes 2022-04-10T01:22:37.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 1619756032 bytes 2022-04-10T01:22:37.659 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 1676915712 bytes 2022-04-10T01:22:37.813 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 1717264384 bytes 2022-04-10T01:22:37.947 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 1747930112 bytes 2022-04-10T01:22:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 1773324288 bytes 2022-04-10T01:22:38.261 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 1846299648 bytes 2022-04-10T01:22:38.401 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 1878999040 bytes 2022-04-10T01:22:38.541 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 1916643328 bytes 2022-04-10T01:22:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 1932438528 bytes 2022-04-10T01:22:38.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 1982396416 bytes 2022-04-10T01:22:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 2083667968 bytes 2022-04-10T01:22:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5400 IOs, 2112976896 bytes 2022-04-10T01:22:39.310 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5500 IOs, 2131163136 bytes 2022-04-10T01:22:39.464 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5600 IOs, 2177842176 bytes 2022-04-10T01:22:39.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5700 IOs, 2223966208 bytes 2022-04-10T01:22:39.789 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5800 IOs, 2255158272 bytes 2022-04-10T01:22:39.947 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5900 IOs, 2304694272 bytes 2022-04-10T01:22:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6000 IOs, 2355588096 bytes 2022-04-10T01:22:40.329 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6100 IOs, 2458493952 bytes 2022-04-10T01:22:40.500 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6200 IOs, 2516689920 bytes 2022-04-10T01:22:40.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6300 IOs, 2577280000 bytes 2022-04-10T01:22:40.834 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6400 IOs, 2622846976 bytes 2022-04-10T01:22:41.005 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6500 IOs, 2688455680 bytes 2022-04-10T01:22:41.107 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6600 IOs, 2689067008 bytes 2022-04-10T01:22:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6700 IOs, 2713212928 bytes 2022-04-10T01:22:41.425 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6800 IOs, 2791461888 bytes 2022-04-10T01:22:41.631 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6900 IOs, 2869485568 bytes 2022-04-10T01:22:41.739 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7000 IOs, 2870113280 bytes 2022-04-10T01:22:41.839 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7100 IOs, 2870721536 bytes 2022-04-10T01:22:41.991 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7200 IOs, 2917858304 bytes 2022-04-10T01:22:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7300 IOs, 2963965952 bytes 2022-04-10T01:22:42.317 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7400 IOs, 3024012288 bytes 2022-04-10T01:22:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7500 IOs, 3062084608 bytes 2022-04-10T01:22:42.659 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7600 IOs, 3133415424 bytes 2022-04-10T01:22:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7700 IOs, 3158096896 bytes 2022-04-10T01:22:42.964 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7800 IOs, 3210143744 bytes 2022-04-10T01:22:43.112 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7900 IOs, 3260060672 bytes 2022-04-10T01:22:43.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:22:43.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:22:43.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:22:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:22:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2022-04-10T01:22:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:22:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:22:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:22:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state_pattern=up+replaying 2022-04-10T01:22:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local description_pattern= 2022-04-10T01:22:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local service_pattern= 2022-04-10T01:22:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:22:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:22:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:22:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:22:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:22:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:22:44.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2022-04-10T01:22:44.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:22:44.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:22:44.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:22:44.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state_pattern=up+replaying 2022-04-10T01:22:44.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local description_pattern= 2022-04-10T01:22:44.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local service_pattern= 2022-04-10T01:22:44.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:22:44.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:22:44.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:22:44.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:22:44.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-test.mirror_status 2022-04-10T01:22:44.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-test.mirror_status 2022-04-10T01:22:44.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:22:44.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:22:44.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ CEPH_ARGS= 2022-04-10T01:22:44.242 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2022-04-10T01:22:44.242 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ tee /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:22:44.289 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:test: 2022-04-10T01:22:44.289 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: global_id: 5ff848f4-a6a0-498d-b9f1-7a041de7de98 2022-04-10T01:22:44.289 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: state: up+replaying 2022-04-10T01:22:44.290 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: description: replaying, {"bytes_per_second":17842756.27,"bytes_per_snapshot":528983040.0,"local_snapshot_timestamp":1649553742,"remote_snapshot_timestamp":1649553742,"replay_state":"idle"} 2022-04-10T01:22:44.290 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: service: mirror.0 on smithi043 2022-04-10T01:22:44.290 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: last_update: 2022-04-10 01:22:39 2022-04-10T01:22:44.291 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: peer_sites: 2022-04-10T01:22:44.291 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: name: cluster2 2022-04-10T01:22:44.291 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: state: up+stopped 2022-04-10T01:22:44.291 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: description: local image is primary 2022-04-10T01:22:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: last_update: 2022-04-10 01:22:39 2022-04-10T01:22:44.324 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:22:44.325 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: state: up+replaying 2022-04-10T01:22:44.326 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep '^ description: .*' /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:22:44.326 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: description: replaying, {"bytes_per_second":17842756.27,"bytes_per_snapshot":528983040.0,"local_snapshot_timestamp":1649553742,"remote_snapshot_timestamp":1649553742,"replay_state":"idle"} 2022-04-10T01:22:44.327 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:22:44.327 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo up+replaying 2022-04-10T01:22:44.328 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep '^up+' 2022-04-10T01:22:44.328 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:up+replaying 2022-04-10T01:22:44.329 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:22:44.329 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: service: mirror.0 on smithi043 2022-04-10T01:22:44.330 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:22:44.332 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local 'last_update=2022-04-10 01:22:39' 2022-04-10T01:22:44.332 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2022-04-10 01:22:39' 2022-04-10T01:22:44.332 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:22:44.332 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:22:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:22:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state_pattern=up+replaying 2022-04-10T01:22:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local 'prev_last_update=2022-04-10 01:22:39' 2022-04-10T01:22:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:22:44.334 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:22:44.334 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:22:44.334 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:22:44.334 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror.mirror_status 2022-04-10T01:22:44.335 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror.mirror_status 2022-04-10T01:22:44.335 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:22:44.336 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror.mirror_status 2022-04-10T01:22:44.336 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2022-04-10T01:22:44.369 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: failed to query services: (13) Permission denied 2022-04-10T01:22:44.400 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local last_update state 2022-04-10T01:22:44.401 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2022-04-10T01:22:44.403 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ last_update='2022-04-10 01:22:39' 2022-04-10T01:22:44.403 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2022-04-10T01:22:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ state=up+replaying 2022-04-10T01:22:44.405 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo up+replaying 2022-04-10T01:22:44.405 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep up+replaying 2022-04-10T01:22:44.406 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:up+replaying 2022-04-10T01:22:44.407 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:22:44.407 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:22:44.408 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ snap_name=snap6 2022-04-10T01:22:44.408 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror test snap6 2022-04-10T01:22:44.408 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:22:44.408 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:22:44.409 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:22:44.409 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap6 2022-04-10T01:22:44.409 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:23:00.783 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.40681811126391254) > weight (0.25) 2022-04-10T01:23:00.783 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.6557099305810371) > weight (0.25) 2022-04-10T01:23:00.784 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2022-04-10T01:23:00.784 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2022-04-10T01:23:00.785 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2022-04-10T01:23:00.785 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2022-04-10T01:23:00.785 INFO:teuthology.orchestra.run:waiting for 600 2022-04-10T01:23:00.786 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi043.stderr:2022-04-10T01:23:00.784+0000 7ff60395e700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 43002) UID: 1000 2022-04-10T01:23:00.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi043.stderr:daemon-helper: command failed with exit status 1 2022-04-10T01:23:06.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-04-10T01:23:06.887 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2022-04-10T01:23:06.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2022-04-10T01:23:06.888 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2022-04-10T01:23:06.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2022-04-10T01:23:06.891 INFO:tasks.rbd_mirror_thrash:waiting for 64 secs before thrashing 2022-04-10T01:23:13.228 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror test 2022-04-10T01:23:13.229 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:23:13.229 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:23:13.229 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:23:13.230 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:23:13.230 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:23:13.230 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:23:13.230 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:23:13.231 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2022-04-10T01:23:13.231 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:23:13.231 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:23:13.231 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:23:13.231 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:23:13.232 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:23:13.232 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:23:13.232 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:23:13.232 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:23:13.233 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:23:13.233 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:23:13.233 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:23:14.232 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror test started 2022-04-10T01:23:14.232 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:23:14.233 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:23:14.233 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:23:14.233 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:23:14.233 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:23:14.234 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:23:14.234 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:23:14.234 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:23:14.234 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:23:14.234 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:23:14.235 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2022-04-10T01:23:14.235 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:23:14.235 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:23:14.235 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:23:14.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:23:14.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:23:14.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:23:14.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:23:14.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:23:14.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:23:14.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:23:14.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:23:14.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:23:14.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:23:14.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:23:14.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:23:14.242 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2022-04-10T01:23:14.242 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:23:14.242 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:23:14.242 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:23:14.243 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:23:14.243 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:23:14.243 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:23:14.243 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:23:14.243 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:23:14.244 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:23:14.244 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:23:14.244 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:23:14.244 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:23:14.245 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:23:14.245 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:23:14.245 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:23:14.245 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:23:14.245 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:23:14.246 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:23:14.246 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:23:14.246 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:23:14.246 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:23:14.247 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:23:14.247 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:23:14.247 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:23:14.247 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:23:14.247 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:23:14.248 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:23:14.248 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:23:14.248 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:23:14.248 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:23:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:23:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:23:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:23:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:23:14.250 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:23:14.250 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:23:14.250 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:23:14.250 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:23:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:23:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:23:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:23:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2022-04-10T01:23:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:23:14.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:23:14.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:23:14.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:23:14.368 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:23:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:23:14.373 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:23:14.373 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:23:14.373 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2022-04-10T01:23:14.374 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:23:14.374 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:23:14.374 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:23:14.374 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:23:14.375 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:23:14.375 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:23:14.375 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:23:14.375 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:23:14.375 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:23:14.376 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:23:14.376 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2022-04-10T01:23:14.376 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:23:14.377 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:23:14.377 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:23:14.377 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:23:14.377 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:23:14.378 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:23:14.378 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:23:14.378 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:23:14.378 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:23:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:23:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:23:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:23:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-test.status 2022-04-10T01:23:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-test.status 2022-04-10T01:23:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:23:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.status 2022-04-10T01:23:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-test.status 2022-04-10T01:23:14.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-test.status 2022-04-10T01:23:14.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:23:14.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:23:14.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror test 2022-04-10T01:23:14.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:23:14.382 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:23:14.382 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:23:14.382 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:23:14.382 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:23:14.383 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:23:14.383 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:23:14.383 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2022-04-10T01:23:15.234 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 26 2022-04-10T01:23:15.248 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.rgwz74WspX/cluster2-mirror-test.status 2022-04-10T01:23:15.248 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:23:15.248 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:23:15.248 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:23:15.249 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.status 2022-04-10T01:23:15.249 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:23:15.249 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:23:15.249 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:23:15.250 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:23:15.250 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2022-04-10T01:23:15.250 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:23:15.292 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:23:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=26 2022-04-10T01:23:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:23:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:23:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:23:15.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:23:15.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:23:15.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:23:15.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:23:15.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:23:15.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:23:15.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:23:15.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:23:15.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:23:15.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2022-04-10T01:23:15.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:23:15.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:23:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=22 2022-04-10T01:23:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 26 = 22 2022-04-10T01:23:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:23:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:23:15.949 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:23:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:23:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:23:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:23:15.951 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:23:15.951 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:23:15.951 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:23:15.951 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:23:15.952 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:23:15.952 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2022-04-10T01:23:15.952 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:23:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:23:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:23:16.001 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:23:16.001 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:23:16.002 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:23:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:23:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 26 = '' 2022-04-10T01:23:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:23:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:23:16.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:23:16.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:23:16.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:23:16.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:23:16.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:23:16.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:23:16.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:23:16.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:23:16.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:23:16.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2022-04-10T01:23:16.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:23:16.856 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:23:16.856 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:23:16.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:23:16.858 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:23:16.858 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:23:16.859 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:23:16.859 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 26 = '' 2022-04-10T01:23:16.859 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:23:16.859 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1.6 2022-04-10T01:23:18.460 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:23:18.460 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:23:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:23:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:23:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:23:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:23:18.463 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:23:18.463 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:23:18.464 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:23:18.464 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2022-04-10T01:23:18.465 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:23:18.530 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:23:18.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:23:18.532 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:23:18.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:23:18.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:23:18.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:23:18.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 26 = '' 2022-04-10T01:23:18.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:23:18.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 2 2022-04-10T01:23:20.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:23:20.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:23:20.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:23:20.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:23:20.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:23:20.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:23:20.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:23:20.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:23:20.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:23:20.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2022-04-10T01:23:20.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:23:20.585 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:23:20.588 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=26 2022-04-10T01:23:20.588 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 26 = 26 2022-04-10T01:23:20.588 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:23:20.589 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror test snap6 2022-04-10T01:23:20.589 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:23:20.589 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:23:20.589 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:23:20.589 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap6 2022-04-10T01:23:20.590 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:23:20.590 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:23:20.590 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:23:20.590 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:23:20.591 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:23:20.591 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:23:20.591 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:23:21.589 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/test@snap6 2022-04-10T01:23:21.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'test': 2022-04-10T01:23:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 512 MiB in 128 objects 2022-04-10T01:23:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:23:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 12 2022-04-10T01:23:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 10ae4a893004 2022-04-10T01:23:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.10ae4a893004 2022-04-10T01:23:21.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:23:21.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:23:21.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:23:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:23:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:18:11 2022 2022-04-10T01:23:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:21:34 2022 2022-04-10T01:23:21.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:18:11 2022 2022-04-10T01:23:21.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:23:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:23:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:23:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 5ff848f4-a6a0-498d-b9f1-7a041de7de98 2022-04-10T01:23:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:23:21.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:23:21.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snap5-clean ']' 2022-04-10T01:23:21.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror test snap5-clean 2022-04-10T01:23:21.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:23:21.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:23:21.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap5-clean 2022-04-10T01:23:21.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:23:21.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:23:21.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:23:21.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:23:21.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:23:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:23:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:23:23.053 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:23:23.056 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:23:23.311 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:23:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:23:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror test snap6 2022-04-10T01:23:23.457 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:23:23.457 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:23:23.457 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap6 2022-04-10T01:23:23.457 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:23:23.458 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:23:23.458 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:23:23.458 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:23:23.458 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:23:24.061 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:23:24.065 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:23:24.848 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:23:24.851 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:23:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:23:25.249 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:23:25.250 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ clean_snap_name=snap6-clean 2022-04-10T01:23:25.250 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror test snap6-clean 2022-04-10T01:23:25.250 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:23:25.251 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:23:25.251 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:23:25.251 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap6-clean 2022-04-10T01:23:25.251 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6-clean --debug-rbd=20 --debug-journaler=20 2022-04-10T01:23:26.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 10` 2022-04-10T01:23:26.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror test 2022-04-10T01:23:26.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:23:26.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:23:26.207 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:23:26.207 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:23:26.209 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=10 2022-04-10T01:23:26.209 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:23:26.209 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:23:26.209 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:23:26.209 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:23:26.210 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:23:26.210 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2022-04-10T01:23:26.712 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 51615744 bytes 2022-04-10T01:23:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 110840832 bytes 2022-04-10T01:23:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 111454208 bytes 2022-04-10T01:23:27.181 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 157755392 bytes 2022-04-10T01:23:27.324 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 193773568 bytes 2022-04-10T01:23:27.465 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 233789440 bytes 2022-04-10T01:23:27.604 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 276206592 bytes 2022-04-10T01:23:27.796 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 339306496 bytes 2022-04-10T01:23:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 339922944 bytes 2022-04-10T01:23:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 363658240 bytes 2022-04-10T01:23:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 429622272 bytes 2022-04-10T01:23:28.417 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 493313024 bytes 2022-04-10T01:23:28.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 512240640 bytes 2022-04-10T01:23:28.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 522520576 bytes 2022-04-10T01:23:28.828 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 573368320 bytes 2022-04-10T01:23:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 625882112 bytes 2022-04-10T01:23:29.151 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 661596160 bytes 2022-04-10T01:23:29.325 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 704073728 bytes 2022-04-10T01:23:29.490 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 751826944 bytes 2022-04-10T01:23:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 776010752 bytes 2022-04-10T01:23:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 794608640 bytes 2022-04-10T01:23:29.901 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 817977344 bytes 2022-04-10T01:23:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 901083136 bytes 2022-04-10T01:23:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 924016640 bytes 2022-04-10T01:23:30.432 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 996957184 bytes 2022-04-10T01:23:30.586 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1038604288 bytes 2022-04-10T01:23:30.719 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1064188928 bytes 2022-04-10T01:23:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1100409856 bytes 2022-04-10T01:23:31.013 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1119636480 bytes 2022-04-10T01:23:31.171 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1167042560 bytes 2022-04-10T01:23:31.361 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1236726784 bytes 2022-04-10T01:23:31.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1274318848 bytes 2022-04-10T01:23:31.734 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1362936832 bytes 2022-04-10T01:23:31.897 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1399088128 bytes 2022-04-10T01:23:32.078 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1457861632 bytes 2022-04-10T01:23:32.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1502743552 bytes 2022-04-10T01:23:32.410 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1561330688 bytes 2022-04-10T01:23:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1585687552 bytes 2022-04-10T01:23:32.788 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1704184832 bytes 2022-04-10T01:23:32.946 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1744877568 bytes 2022-04-10T01:23:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1793129472 bytes 2022-04-10T01:23:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 1857049600 bytes 2022-04-10T01:23:33.464 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 1914041344 bytes 2022-04-10T01:23:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 1983951872 bytes 2022-04-10T01:23:33.810 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 2021065728 bytes 2022-04-10T01:23:33.986 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 2069623808 bytes 2022-04-10T01:23:34.323 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 2162414592 bytes 2022-04-10T01:23:34.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 2225387520 bytes 2022-04-10T01:23:34.677 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 2262667264 bytes 2022-04-10T01:23:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 2270452736 bytes 2022-04-10T01:23:34.955 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 2302065664 bytes 2022-04-10T01:23:35.130 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 2360151040 bytes 2022-04-10T01:23:35.299 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 2417783808 bytes 2022-04-10T01:23:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5400 IOs, 2471944192 bytes 2022-04-10T01:23:35.609 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5500 IOs, 2487866368 bytes 2022-04-10T01:23:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5600 IOs, 2533909504 bytes 2022-04-10T01:23:35.949 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5700 IOs, 2584975360 bytes 2022-04-10T01:23:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5800 IOs, 2604555264 bytes 2022-04-10T01:23:36.216 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:23:36.217 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:23:36.217 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:23:36.217 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:23:36.217 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2022-04-10T01:23:36.218 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:23:36.218 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:23:36.218 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:23:36.218 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state_pattern=up+replaying 2022-04-10T01:23:36.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local description_pattern= 2022-04-10T01:23:36.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local service_pattern= 2022-04-10T01:23:36.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:23:36.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:23:36.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:23:36.220 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:23:36.220 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:23:36.220 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:23:37.218 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2022-04-10T01:23:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:23:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:23:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:23:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state_pattern=up+replaying 2022-04-10T01:23:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local description_pattern= 2022-04-10T01:23:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local service_pattern= 2022-04-10T01:23:37.221 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:23:37.221 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:23:37.221 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:23:37.222 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:23:37.222 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-test.mirror_status 2022-04-10T01:23:37.223 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-test.mirror_status 2022-04-10T01:23:37.223 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:23:37.224 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:23:37.224 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ CEPH_ARGS= 2022-04-10T01:23:37.225 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2022-04-10T01:23:37.225 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ tee /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:23:37.270 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:test: 2022-04-10T01:23:37.270 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: global_id: 5ff848f4-a6a0-498d-b9f1-7a041de7de98 2022-04-10T01:23:37.271 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: state: up+replaying 2022-04-10T01:23:37.271 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":528983040.0,"local_snapshot_timestamp":1649553742,"remote_snapshot_timestamp":1649553742,"replay_state":"idle"} 2022-04-10T01:23:37.271 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: service: mirror.0 on smithi043 2022-04-10T01:23:37.271 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: last_update: 2022-04-10 01:23:09 2022-04-10T01:23:37.271 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: peer_sites: 2022-04-10T01:23:37.272 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: name: cluster2 2022-04-10T01:23:37.272 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: state: up+stopped 2022-04-10T01:23:37.272 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: description: local image is primary 2022-04-10T01:23:37.272 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: last_update: 2022-04-10 01:23:09 2022-04-10T01:23:37.274 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:23:37.275 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: state: up+replaying 2022-04-10T01:23:37.276 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep '^ description: .*' /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:23:37.276 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":528983040.0,"local_snapshot_timestamp":1649553742,"remote_snapshot_timestamp":1649553742,"replay_state":"idle"} 2022-04-10T01:23:37.277 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:23:37.277 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo up+replaying 2022-04-10T01:23:37.278 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep '^up+' 2022-04-10T01:23:37.278 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:up+replaying 2022-04-10T01:23:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:23:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: service: mirror.0 on smithi043 2022-04-10T01:23:37.280 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:23:37.281 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local 'last_update=2022-04-10 01:23:09' 2022-04-10T01:23:37.282 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2022-04-10 01:23:09' 2022-04-10T01:23:37.282 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:23:37.282 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:23:37.283 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:23:37.283 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state_pattern=up+replaying 2022-04-10T01:23:37.283 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local 'prev_last_update=2022-04-10 01:23:09' 2022-04-10T01:23:37.283 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:23:37.284 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:23:37.284 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:23:37.284 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:23:37.284 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror.mirror_status 2022-04-10T01:23:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror.mirror_status 2022-04-10T01:23:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:23:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror.mirror_status 2022-04-10T01:23:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2022-04-10T01:23:37.316 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: failed to query services: (13) Permission denied 2022-04-10T01:23:37.339 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local last_update state 2022-04-10T01:23:37.339 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2022-04-10T01:23:37.341 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ last_update='2022-04-10 01:23:09' 2022-04-10T01:23:37.341 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2022-04-10T01:23:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ state=up+replaying 2022-04-10T01:23:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo up+replaying 2022-04-10T01:23:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep up+replaying 2022-04-10T01:23:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:up+replaying 2022-04-10T01:23:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:23:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:23:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ snap_name=snap7 2022-04-10T01:23:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror test snap7 2022-04-10T01:23:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:23:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:23:37.347 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:23:37.347 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap7 2022-04-10T01:23:37.347 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:24:05.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror test 2022-04-10T01:24:05.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:24:05.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:24:05.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:24:05.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:24:05.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:24:05.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:24:05.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:24:05.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2022-04-10T01:24:05.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:24:05.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:24:05.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:24:05.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:24:05.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:24:05.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:24:05.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:24:05.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:24:05.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:24:05.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:24:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:24:06.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror test started 2022-04-10T01:24:06.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:24:06.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:24:06.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:24:06.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:24:06.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:24:06.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:24:06.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:24:06.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:24:06.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:24:06.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:24:06.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2022-04-10T01:24:06.242 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:24:06.242 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:24:06.242 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:24:06.242 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:24:06.243 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:24:06.243 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:24:06.243 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:24:06.243 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:24:06.244 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:24:06.244 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:24:06.244 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:24:06.244 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:24:06.245 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:24:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:24:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:24:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2022-04-10T01:24:06.250 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:24:06.250 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:24:06.250 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:24:06.250 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:24:06.251 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:24:06.251 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:24:06.251 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:24:06.251 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:24:06.252 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:24:06.252 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:24:06.252 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:24:06.252 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:24:06.252 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:24:06.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:24:06.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:24:06.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:24:06.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:24:06.254 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:24:06.254 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:24:06.254 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:24:06.254 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:24:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:24:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:24:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:24:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:24:06.256 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:24:06.256 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:24:06.256 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:24:06.256 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:24:06.256 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:24:06.257 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:24:06.257 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:24:06.257 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:24:06.257 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:24:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:24:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:24:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:24:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:24:06.260 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:24:06.261 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:24:06.261 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:24:06.261 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2022-04-10T01:24:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:24:06.374 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:24:06.375 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:24:06.375 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:24:06.375 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:24:06.380 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:24:06.380 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:24:06.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:24:06.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2022-04-10T01:24:06.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:24:06.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:24:06.382 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:24:06.382 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:24:06.382 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:24:06.383 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:24:06.383 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:24:06.383 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:24:06.383 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:24:06.383 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:24:06.384 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2022-04-10T01:24:06.384 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:24:06.384 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:24:06.384 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:24:06.385 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:24:06.385 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:24:06.385 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:24:06.385 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:24:06.385 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:24:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:24:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:24:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:24:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:24:06.387 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-test.status 2022-04-10T01:24:06.387 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-test.status 2022-04-10T01:24:06.387 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:24:06.387 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.status 2022-04-10T01:24:06.388 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-test.status 2022-04-10T01:24:06.388 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-test.status 2022-04-10T01:24:06.388 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:24:06.388 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:24:06.389 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror test 2022-04-10T01:24:06.389 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:24:06.389 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:24:06.389 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:24:06.389 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:24:06.390 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:24:06.390 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:24:06.390 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:24:06.390 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2022-04-10T01:24:07.242 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 30 2022-04-10T01:24:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.rgwz74WspX/cluster2-mirror-test.status 2022-04-10T01:24:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:24:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:24:07.254 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:24:07.254 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.status 2022-04-10T01:24:07.254 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:24:07.254 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:24:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:24:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:24:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2022-04-10T01:24:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:24:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:24:07.300 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=30 2022-04-10T01:24:07.300 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:24:07.300 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:24:07.301 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:24:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:24:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:24:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:24:07.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:24:07.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:24:07.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:24:07.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:24:07.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:24:07.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:24:07.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2022-04-10T01:24:07.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:24:07.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:24:07.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=26 2022-04-10T01:24:07.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 30 = 26 2022-04-10T01:24:07.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:24:07.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:24:07.953 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:24:07.953 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:24:07.953 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:24:07.953 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:24:07.954 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:24:07.954 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:24:07.954 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:24:07.954 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:24:07.955 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:24:07.955 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2022-04-10T01:24:07.955 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:24:08.001 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:24:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:24:08.003 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:24:08.003 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:24:08.003 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:24:08.004 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:24:08.004 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 30 = '' 2022-04-10T01:24:08.005 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:24:08.005 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:24:08.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:24:08.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:24:08.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:24:08.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:24:08.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:24:08.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:24:08.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:24:08.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:24:08.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:24:08.809 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2022-04-10T01:24:08.809 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:24:08.856 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:24:08.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:24:08.858 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:24:08.858 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:24:08.859 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:24:08.860 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:24:08.860 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 30 = '' 2022-04-10T01:24:08.860 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:24:08.860 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1.6 2022-04-10T01:24:10.460 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:24:10.461 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:24:10.461 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:24:10.461 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:24:10.462 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:24:10.462 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:24:10.462 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:24:10.462 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:24:10.463 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:24:10.463 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2022-04-10T01:24:10.463 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:24:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:24:10.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=30 2022-04-10T01:24:10.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 30 = 30 2022-04-10T01:24:10.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:24:10.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror test snap7 2022-04-10T01:24:10.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:24:10.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:24:10.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:24:10.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap7 2022-04-10T01:24:10.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:24:10.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:24:10.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:24:10.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:24:10.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:24:10.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:24:10.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:24:10.892 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6243414472406202) > weight (0.25) 2022-04-10T01:24:10.892 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5786369072565383) > weight (0.25) 2022-04-10T01:24:10.893 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.6885830890430057) > weight (0.25) 2022-04-10T01:24:10.893 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.698313859194782) > weight (0.25) 2022-04-10T01:24:10.893 INFO:tasks.rbd_mirror_thrash:waiting for 111 secs before thrashing 2022-04-10T01:24:11.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/test@snap7 2022-04-10T01:24:11.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'test': 2022-04-10T01:24:11.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 512 MiB in 128 objects 2022-04-10T01:24:11.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:24:11.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 15 2022-04-10T01:24:11.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 10ae4a893004 2022-04-10T01:24:11.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.10ae4a893004 2022-04-10T01:24:11.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:24:11.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, operations, non-primary 2022-04-10T01:24:11.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: snap-trash 2022-04-10T01:24:11.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:24:11.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:18:11 2022 2022-04-10T01:24:11.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:23:22 2022 2022-04-10T01:24:11.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:18:11 2022 2022-04-10T01:24:11.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:24:11.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:24:11.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:24:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 5ff848f4-a6a0-498d-b9f1-7a041de7de98 2022-04-10T01:24:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:24:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:24:11.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snap6-clean ']' 2022-04-10T01:24:11.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror test snap6-clean 2022-04-10T01:24:11.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:24:11.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:24:11.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap6-clean 2022-04-10T01:24:11.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:24:11.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:24:11.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:24:11.556 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:24:11.556 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:24:12.131 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:24:12.135 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:24:12.914 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:24:12.917 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:24:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:24:13.313 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:24:13.313 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror test snap7 2022-04-10T01:24:13.314 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:24:13.314 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:24:13.314 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap7 2022-04-10T01:24:13.314 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:24:13.315 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:24:13.315 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:24:13.315 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:24:13.315 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:24:14.313 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:24:14.316 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:24:15.123 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:24:15.127 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:24:15.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:24:15.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:24:15.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ clean_snap_name=snap7-clean 2022-04-10T01:24:15.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror test snap7-clean 2022-04-10T01:24:15.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:24:15.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:24:15.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:24:15.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap7-clean 2022-04-10T01:24:15.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7-clean --debug-rbd=20 --debug-journaler=20 2022-04-10T01:24:16.248 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 10` 2022-04-10T01:24:16.249 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror test 2022-04-10T01:24:16.249 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:24:16.249 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:24:16.250 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:24:16.250 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:24:16.251 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=12 2022-04-10T01:24:16.251 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:24:16.251 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:24:16.252 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:24:16.252 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:24:16.252 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:24:16.252 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2022-04-10T01:24:16.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 54857728 bytes 2022-04-10T01:24:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 101091328 bytes 2022-04-10T01:24:16.928 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 156938240 bytes 2022-04-10T01:24:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 198418432 bytes 2022-04-10T01:24:17.234 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 214471680 bytes 2022-04-10T01:24:17.416 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 270948352 bytes 2022-04-10T01:24:17.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 285032448 bytes 2022-04-10T01:24:17.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 313325568 bytes 2022-04-10T01:24:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 335066112 bytes 2022-04-10T01:24:17.977 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 375068672 bytes 2022-04-10T01:24:18.094 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 375690240 bytes 2022-04-10T01:24:18.235 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 395219968 bytes 2022-04-10T01:24:18.362 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 410010624 bytes 2022-04-10T01:24:18.562 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 485978112 bytes 2022-04-10T01:24:18.715 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 525735936 bytes 2022-04-10T01:24:18.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 564524032 bytes 2022-04-10T01:24:19.069 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 644129792 bytes 2022-04-10T01:24:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 736729088 bytes 2022-04-10T01:24:19.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 801629184 bytes 2022-04-10T01:24:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 856675328 bytes 2022-04-10T01:24:19.915 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 939928576 bytes 2022-04-10T01:24:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 998726656 bytes 2022-04-10T01:24:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 1085025280 bytes 2022-04-10T01:24:20.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 1172727808 bytes 2022-04-10T01:24:20.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 1206792192 bytes 2022-04-10T01:24:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1251540992 bytes 2022-04-10T01:24:21.002 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1281362944 bytes 2022-04-10T01:24:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1342092288 bytes 2022-04-10T01:24:21.328 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1353740288 bytes 2022-04-10T01:24:21.454 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1367629824 bytes 2022-04-10T01:24:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1457230848 bytes 2022-04-10T01:24:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1492348928 bytes 2022-04-10T01:24:21.954 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1502217216 bytes 2022-04-10T01:24:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1502832640 bytes 2022-04-10T01:24:22.306 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1599171584 bytes 2022-04-10T01:24:22.489 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1654614016 bytes 2022-04-10T01:24:22.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1697582080 bytes 2022-04-10T01:24:22.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1736327168 bytes 2022-04-10T01:24:22.978 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1776588800 bytes 2022-04-10T01:24:23.151 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1825443840 bytes 2022-04-10T01:24:23.305 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1857174528 bytes 2022-04-10T01:24:23.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 1929166848 bytes 2022-04-10T01:24:23.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 1971162112 bytes 2022-04-10T01:24:23.790 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.31221419017481167) > weight (0.25) 2022-04-10T01:24:23.790 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.96241148853267) > weight (0.25) 2022-04-10T01:24:23.791 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.5263467977872835) > weight (0.25) 2022-04-10T01:24:23.791 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.6965112161822008) > weight (0.25) 2022-04-10T01:24:23.791 INFO:tasks.rbd_mirror_thrash:waiting for 107 secs before thrashing 2022-04-10T01:24:23.906 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 2076558336 bytes 2022-04-10T01:24:24.059 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 2099781632 bytes 2022-04-10T01:24:24.246 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 2157266944 bytes 2022-04-10T01:24:24.408 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 2196161536 bytes 2022-04-10T01:24:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 2221954048 bytes 2022-04-10T01:24:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 2222569472 bytes 2022-04-10T01:24:24.880 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 2299926528 bytes 2022-04-10T01:24:24.998 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 2300562432 bytes 2022-04-10T01:24:25.148 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 2323866624 bytes 2022-04-10T01:24:25.307 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 2367870976 bytes 2022-04-10T01:24:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5400 IOs, 2431842304 bytes 2022-04-10T01:24:25.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5500 IOs, 2449826816 bytes 2022-04-10T01:24:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5600 IOs, 2474309632 bytes 2022-04-10T01:24:25.928 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5700 IOs, 2487954432 bytes 2022-04-10T01:24:26.072 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5800 IOs, 2516089856 bytes 2022-04-10T01:24:26.260 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5900 IOs, 2581035008 bytes 2022-04-10T01:24:26.441 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6000 IOs, 2631842816 bytes 2022-04-10T01:24:26.604 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6100 IOs, 2669386752 bytes 2022-04-10T01:24:26.827 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6200 IOs, 2767316992 bytes 2022-04-10T01:24:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6300 IOs, 2836571136 bytes 2022-04-10T01:24:27.186 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6400 IOs, 2861190144 bytes 2022-04-10T01:24:27.333 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6500 IOs, 2888654848 bytes 2022-04-10T01:24:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6600 IOs, 2921865216 bytes 2022-04-10T01:24:27.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6700 IOs, 2960795648 bytes 2022-04-10T01:24:27.799 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6800 IOs, 2994177024 bytes 2022-04-10T01:24:28.002 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6900 IOs, 3069377536 bytes 2022-04-10T01:24:28.200 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7000 IOs, 3122509824 bytes 2022-04-10T01:24:28.258 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:24:28.258 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:24:28.259 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:24:28.259 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:24:28.259 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2022-04-10T01:24:28.259 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:24:28.260 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:24:28.260 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:24:28.260 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state_pattern=up+replaying 2022-04-10T01:24:28.260 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local description_pattern= 2022-04-10T01:24:28.261 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local service_pattern= 2022-04-10T01:24:28.261 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:24:28.261 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:24:28.261 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:24:28.261 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:24:28.262 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:24:28.262 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:24:29.260 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2022-04-10T01:24:29.261 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:24:29.261 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:24:29.261 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:24:29.262 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state_pattern=up+replaying 2022-04-10T01:24:29.262 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local description_pattern= 2022-04-10T01:24:29.262 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local service_pattern= 2022-04-10T01:24:29.262 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:24:29.263 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:24:29.263 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:24:29.263 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:24:29.263 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-test.mirror_status 2022-04-10T01:24:29.264 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-test.mirror_status 2022-04-10T01:24:29.264 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:24:29.265 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:24:29.266 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ CEPH_ARGS= 2022-04-10T01:24:29.266 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2022-04-10T01:24:29.266 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ tee /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:24:29.315 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:test: 2022-04-10T01:24:29.316 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: global_id: 5ff848f4-a6a0-498d-b9f1-7a041de7de98 2022-04-10T01:24:29.316 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: state: up+replaying 2022-04-10T01:24:29.317 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":534634496.0,"local_snapshot_timestamp":1649553795,"remote_snapshot_timestamp":1649553795,"replay_state":"idle"} 2022-04-10T01:24:29.317 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: service: mirror.0 on smithi043 2022-04-10T01:24:29.317 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: last_update: 2022-04-10 01:24:09 2022-04-10T01:24:29.318 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: peer_sites: 2022-04-10T01:24:29.318 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: name: cluster2 2022-04-10T01:24:29.318 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: state: up+stopped 2022-04-10T01:24:29.318 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: description: local image is primary 2022-04-10T01:24:29.318 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: last_update: 2022-04-10 01:24:09 2022-04-10T01:24:29.320 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:24:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: state: up+replaying 2022-04-10T01:24:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep '^ description: .*' /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:24:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":534634496.0,"local_snapshot_timestamp":1649553795,"remote_snapshot_timestamp":1649553795,"replay_state":"idle"} 2022-04-10T01:24:29.324 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:24:29.324 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo up+replaying 2022-04-10T01:24:29.324 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep '^up+' 2022-04-10T01:24:29.325 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:up+replaying 2022-04-10T01:24:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:24:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: service: mirror.0 on smithi043 2022-04-10T01:24:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:24:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local 'last_update=2022-04-10 01:24:09' 2022-04-10T01:24:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2022-04-10 01:24:09' 2022-04-10T01:24:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:24:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:24:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:24:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state_pattern=up+replaying 2022-04-10T01:24:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local 'prev_last_update=2022-04-10 01:24:09' 2022-04-10T01:24:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:24:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:24:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:24:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:24:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror.mirror_status 2022-04-10T01:24:29.332 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror.mirror_status 2022-04-10T01:24:29.332 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:24:29.333 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror.mirror_status 2022-04-10T01:24:29.333 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2022-04-10T01:24:29.363 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: failed to query services: (13) Permission denied 2022-04-10T01:24:29.389 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local last_update state 2022-04-10T01:24:29.390 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2022-04-10T01:24:29.392 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ last_update='2022-04-10 01:24:09' 2022-04-10T01:24:29.392 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2022-04-10T01:24:29.394 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ state=up+replaying 2022-04-10T01:24:29.394 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo up+replaying 2022-04-10T01:24:29.395 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep up+replaying 2022-04-10T01:24:29.395 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:up+replaying 2022-04-10T01:24:29.396 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:24:29.397 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:24:29.397 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ snap_name=snap8 2022-04-10T01:24:29.397 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror test snap8 2022-04-10T01:24:29.397 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:24:29.398 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:24:29.398 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:24:29.398 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap8 2022-04-10T01:24:29.398 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:24:57.852 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror test 2022-04-10T01:24:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:24:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:24:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:24:57.854 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:24:57.854 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:24:57.854 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:24:57.854 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:24:57.855 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2022-04-10T01:24:57.855 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:24:57.855 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:24:57.855 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:24:57.856 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:24:57.856 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:24:57.856 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:24:57.856 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:24:57.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:24:57.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:24:57.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:24:57.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:24:58.863 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror test started 2022-04-10T01:24:58.864 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:24:58.864 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:24:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:24:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:24:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:24:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:24:58.867 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:24:58.867 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:24:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:24:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:24:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2022-04-10T01:24:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:24:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:24:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:24:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:24:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:24:58.872 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:24:58.872 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:24:58.872 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:24:58.873 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:24:58.873 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:24:58.873 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:24:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:24:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:24:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:24:58.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:24:58.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2022-04-10T01:24:58.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:24:58.876 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:24:58.876 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:24:58.876 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:24:58.876 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:24:58.877 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:24:58.877 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:24:58.877 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:24:58.877 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:24:58.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:24:58.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:24:58.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:24:58.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:24:58.879 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:24:58.879 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:24:58.879 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:24:58.879 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:24:58.880 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:24:58.880 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:24:58.880 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:24:58.880 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:24:58.880 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:24:58.881 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:24:58.881 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:24:58.881 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:24:58.881 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:24:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:24:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:24:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:24:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:24:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:24:58.883 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:24:58.883 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:24:58.883 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:24:58.883 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:24:58.884 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:24:58.884 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:24:58.884 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:24:58.885 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:24:58.885 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:24:58.886 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:24:58.886 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2022-04-10T01:24:58.988 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:24:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:24:58.990 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:24:58.990 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:24:58.991 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:24:58.995 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:24:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:24:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:24:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2022-04-10T01:24:58.997 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:24:58.997 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:24:58.997 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:24:58.997 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:24:58.998 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:24:58.998 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:24:58.998 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:24:58.998 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:24:58.999 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:24:58.999 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:24:58.999 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2022-04-10T01:24:58.999 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:24:59.000 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:24:59.000 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:24:59.000 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:24:59.000 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:24:59.000 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:24:59.001 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:24:59.001 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:24:59.001 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:24:59.001 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:24:59.002 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:24:59.002 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:24:59.002 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-test.status 2022-04-10T01:24:59.002 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-test.status 2022-04-10T01:24:59.003 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:24:59.003 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.status 2022-04-10T01:24:59.003 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-test.status 2022-04-10T01:24:59.003 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-test.status 2022-04-10T01:24:59.004 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:24:59.004 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:24:59.004 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror test 2022-04-10T01:24:59.004 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:24:59.005 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:24:59.005 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:24:59.005 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:24:59.005 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:24:59.005 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:24:59.006 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:24:59.006 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2022-04-10T01:24:59.862 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 34 2022-04-10T01:24:59.872 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.rgwz74WspX/cluster2-mirror-test.status 2022-04-10T01:24:59.873 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:24:59.873 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:24:59.873 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:24:59.873 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.status 2022-04-10T01:24:59.874 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:24:59.874 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:24:59.874 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:24:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:24:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2022-04-10T01:24:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:24:59.918 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:24:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=34 2022-04-10T01:24:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:24:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:24:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:25:00.122 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:25:00.122 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:25:00.122 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:25:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:25:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:25:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:25:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:25:00.124 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:25:00.124 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:25:00.124 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2022-04-10T01:25:00.124 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:25:00.173 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:25:00.175 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=30 2022-04-10T01:25:00.176 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 34 = 30 2022-04-10T01:25:00.176 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:25:00.176 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:25:00.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:25:00.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:25:00.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:25:00.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:25:00.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:25:00.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:25:00.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:25:00.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:25:00.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:25:00.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2022-04-10T01:25:00.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:25:00.628 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:25:00.628 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:25:00.629 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:25:00.629 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:25:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:25:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:25:00.631 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 34 = '' 2022-04-10T01:25:00.631 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:25:00.631 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:25:01.432 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:25:01.432 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:25:01.433 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:25:01.433 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:25:01.433 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:25:01.433 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:25:01.434 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:25:01.434 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:25:01.434 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:25:01.434 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2022-04-10T01:25:01.435 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:25:01.483 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:25:01.484 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:25:01.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:25:01.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:25:01.486 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:25:01.486 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:25:01.487 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 34 = '' 2022-04-10T01:25:01.487 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:25:01.487 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1.6 2022-04-10T01:25:03.088 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:25:03.088 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:25:03.088 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:25:03.088 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:25:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:25:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:25:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:25:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:25:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:25:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2022-04-10T01:25:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:25:03.175 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:25:03.175 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:25:03.177 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:25:03.178 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:25:03.178 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:25:03.179 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:25:03.179 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 34 = '' 2022-04-10T01:25:03.179 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:25:03.179 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 2 2022-04-10T01:25:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:25:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:25:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:25:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:25:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:25:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:25:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:25:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:25:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:25:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2022-04-10T01:25:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:25:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:25:05.235 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=34 2022-04-10T01:25:05.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 34 = 34 2022-04-10T01:25:05.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:25:05.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror test snap8 2022-04-10T01:25:05.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:25:05.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:25:05.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:25:05.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap8 2022-04-10T01:25:05.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:25:05.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:25:05.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:25:05.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:25:05.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:25:05.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:25:05.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:25:06.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/test@snap8 2022-04-10T01:25:06.282 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'test': 2022-04-10T01:25:06.282 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 512 MiB in 128 objects 2022-04-10T01:25:06.282 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:25:06.283 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 16 2022-04-10T01:25:06.283 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 10ae4a893004 2022-04-10T01:25:06.283 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.10ae4a893004 2022-04-10T01:25:06.283 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:25:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:25:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:25:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:25:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:18:11 2022 2022-04-10T01:25:06.285 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:23:22 2022 2022-04-10T01:25:06.285 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:18:11 2022 2022-04-10T01:25:06.285 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:25:06.285 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:25:06.286 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:25:06.286 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 5ff848f4-a6a0-498d-b9f1-7a041de7de98 2022-04-10T01:25:06.286 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:25:06.287 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:25:06.288 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snap7-clean ']' 2022-04-10T01:25:06.288 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror test snap7-clean 2022-04-10T01:25:06.288 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:25:06.288 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:25:06.289 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap7-clean 2022-04-10T01:25:06.289 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:25:06.289 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:25:06.289 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:25:06.290 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:25:06.290 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:25:06.868 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:25:06.871 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:25:07.601 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:25:07.605 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:25:07.859 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:25:08.005 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:25:08.005 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror test snap8 2022-04-10T01:25:08.006 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:25:08.006 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:25:08.006 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap8 2022-04-10T01:25:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:25:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:25:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:25:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:25:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:25:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:25:08.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:25:09.318 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:25:09.322 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:25:09.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:25:09.727 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:25:09.727 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ clean_snap_name=snap8-clean 2022-04-10T01:25:09.727 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror test snap8-clean 2022-04-10T01:25:09.727 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:25:09.728 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:25:09.728 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:25:09.728 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap8-clean 2022-04-10T01:25:09.728 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8-clean --debug-rbd=20 --debug-journaler=20 2022-04-10T01:25:10.242 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 10` 2022-04-10T01:25:10.242 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror test 2022-04-10T01:25:10.243 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:25:10.243 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:25:10.243 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:25:10.243 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:25:10.245 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=14 2022-04-10T01:25:10.245 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:25:10.245 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:25:10.245 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:25:10.245 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:25:10.246 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:25:10.246 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2022-04-10T01:25:10.723 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 66358272 bytes 2022-04-10T01:25:10.923 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 111016960 bytes 2022-04-10T01:25:11.146 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 194789376 bytes 2022-04-10T01:25:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 230267904 bytes 2022-04-10T01:25:11.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 266442752 bytes 2022-04-10T01:25:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 339544064 bytes 2022-04-10T01:25:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 346560512 bytes 2022-04-10T01:25:11.970 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 393065472 bytes 2022-04-10T01:25:12.125 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 421256192 bytes 2022-04-10T01:25:12.314 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 478040064 bytes 2022-04-10T01:25:12.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 538430464 bytes 2022-04-10T01:25:12.681 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 586944512 bytes 2022-04-10T01:25:12.845 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 614761472 bytes 2022-04-10T01:25:12.960 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 615371776 bytes 2022-04-10T01:25:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 680815616 bytes 2022-04-10T01:25:13.311 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 705780736 bytes 2022-04-10T01:25:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 782084096 bytes 2022-04-10T01:25:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 805846016 bytes 2022-04-10T01:25:13.842 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 852701184 bytes 2022-04-10T01:25:13.985 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 875046912 bytes 2022-04-10T01:25:14.142 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 907110400 bytes 2022-04-10T01:25:14.327 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 952996864 bytes 2022-04-10T01:25:14.487 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 984876032 bytes 2022-04-10T01:25:14.647 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 1015213056 bytes 2022-04-10T01:25:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 1058287616 bytes 2022-04-10T01:25:14.974 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1086934016 bytes 2022-04-10T01:25:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1174744064 bytes 2022-04-10T01:25:15.364 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1207780352 bytes 2022-04-10T01:25:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1267720192 bytes 2022-04-10T01:25:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1299197952 bytes 2022-04-10T01:25:15.916 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1372352512 bytes 2022-04-10T01:25:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1422556160 bytes 2022-04-10T01:25:16.224 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1461422080 bytes 2022-04-10T01:25:16.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1554958336 bytes 2022-04-10T01:25:16.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1596770304 bytes 2022-04-10T01:25:16.792 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1614725120 bytes 2022-04-10T01:25:16.959 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1649617920 bytes 2022-04-10T01:25:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1706163200 bytes 2022-04-10T01:25:17.391 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1798835200 bytes 2022-04-10T01:25:17.544 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1821297664 bytes 2022-04-10T01:25:17.677 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1834646528 bytes 2022-04-10T01:25:17.837 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 1863674880 bytes 2022-04-10T01:25:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 1864293376 bytes 2022-04-10T01:25:18.148 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 1922853888 bytes 2022-04-10T01:25:18.273 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 1929636864 bytes 2022-04-10T01:25:18.410 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 1940035584 bytes 2022-04-10T01:25:18.607 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 2007580672 bytes 2022-04-10T01:25:18.781 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 2067675136 bytes 2022-04-10T01:25:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 2086322176 bytes 2022-04-10T01:25:19.101 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 2109188096 bytes 2022-04-10T01:25:19.281 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 2151279616 bytes 2022-04-10T01:25:19.430 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 2170704896 bytes 2022-04-10T01:25:19.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 2192440320 bytes 2022-04-10T01:25:19.796 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5400 IOs, 2271611904 bytes 2022-04-10T01:25:19.947 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5500 IOs, 2292046848 bytes 2022-04-10T01:25:20.073 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5600 IOs, 2301702144 bytes 2022-04-10T01:25:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5700 IOs, 2368573440 bytes 2022-04-10T01:25:20.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5800 IOs, 2428544000 bytes 2022-04-10T01:25:20.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5900 IOs, 2473767936 bytes 2022-04-10T01:25:20.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6000 IOs, 2520040448 bytes 2022-04-10T01:25:20.995 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6100 IOs, 2553125888 bytes 2022-04-10T01:25:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6200 IOs, 2636094464 bytes 2022-04-10T01:25:21.434 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6300 IOs, 2719543296 bytes 2022-04-10T01:25:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6400 IOs, 2756596736 bytes 2022-04-10T01:25:21.748 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6500 IOs, 2769095680 bytes 2022-04-10T01:25:21.994 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6600 IOs, 2872432640 bytes 2022-04-10T01:25:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6700 IOs, 2901590016 bytes 2022-04-10T01:25:22.293 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6800 IOs, 2915660800 bytes 2022-04-10T01:25:22.445 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6900 IOs, 2939850752 bytes 2022-04-10T01:25:22.630 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7000 IOs, 2990583808 bytes 2022-04-10T01:25:22.896 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7100 IOs, 3113021440 bytes 2022-04-10T01:25:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7200 IOs, 3140220928 bytes 2022-04-10T01:25:23.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7300 IOs, 3166427136 bytes 2022-04-10T01:25:23.452 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7400 IOs, 3280797696 bytes 2022-04-10T01:25:23.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7500 IOs, 3340311552 bytes 2022-04-10T01:25:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7600 IOs, 3400945664 bytes 2022-04-10T01:25:24.021 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7700 IOs, 3441193984 bytes 2022-04-10T01:25:24.149 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7800 IOs, 3443582976 bytes 2022-04-10T01:25:24.255 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:25:24.255 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:25:24.256 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:25:24.256 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:25:24.256 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2022-04-10T01:25:24.256 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:25:24.257 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:25:24.257 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:25:24.257 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state_pattern=up+replaying 2022-04-10T01:25:24.257 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local description_pattern= 2022-04-10T01:25:24.257 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local service_pattern= 2022-04-10T01:25:24.258 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:25:24.258 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:25:24.258 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:25:24.258 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:25:24.259 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:25:24.259 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:25:25.257 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2022-04-10T01:25:25.257 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:25:25.257 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:25:25.258 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:25:25.258 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state_pattern=up+replaying 2022-04-10T01:25:25.258 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local description_pattern= 2022-04-10T01:25:25.258 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local service_pattern= 2022-04-10T01:25:25.259 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:25:25.259 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:25:25.259 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:25:25.259 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:25:25.260 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-test.mirror_status 2022-04-10T01:25:25.260 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-test.mirror_status 2022-04-10T01:25:25.260 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:25:25.261 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:25:25.261 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ CEPH_ARGS= 2022-04-10T01:25:25.261 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2022-04-10T01:25:25.262 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ tee /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:25:25.310 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:test: 2022-04-10T01:25:25.310 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: global_id: 5ff848f4-a6a0-498d-b9f1-7a041de7de98 2022-04-10T01:25:25.310 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: state: up+replaying 2022-04-10T01:25:25.311 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: description: replaying, {"bytes_per_second":17882009.6,"bytes_per_snapshot":534991360.0,"local_snapshot_timestamp":1649553899,"remote_snapshot_timestamp":1649553899,"replay_state":"idle"} 2022-04-10T01:25:25.311 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: service: mirror.0 on smithi043 2022-04-10T01:25:25.311 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: last_update: 2022-04-10 01:25:09 2022-04-10T01:25:25.312 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: peer_sites: 2022-04-10T01:25:25.312 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: name: cluster2 2022-04-10T01:25:25.312 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: state: up+stopped 2022-04-10T01:25:25.312 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: description: local image is primary 2022-04-10T01:25:25.313 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: last_update: 2022-04-10 01:25:09 2022-04-10T01:25:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:25:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: state: up+replaying 2022-04-10T01:25:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep '^ description: .*' /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:25:25.317 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: description: replaying, {"bytes_per_second":17882009.6,"bytes_per_snapshot":534991360.0,"local_snapshot_timestamp":1649553899,"remote_snapshot_timestamp":1649553899,"replay_state":"idle"} 2022-04-10T01:25:25.318 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:25:25.318 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo up+replaying 2022-04-10T01:25:25.318 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep '^up+' 2022-04-10T01:25:25.319 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:up+replaying 2022-04-10T01:25:25.319 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:25:25.320 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: service: mirror.0 on smithi043 2022-04-10T01:25:25.320 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:25:25.322 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local 'last_update=2022-04-10 01:25:09' 2022-04-10T01:25:25.322 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2022-04-10 01:25:09' 2022-04-10T01:25:25.322 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:25:25.323 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:25:25.323 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:25:25.323 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state_pattern=up+replaying 2022-04-10T01:25:25.323 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local 'prev_last_update=2022-04-10 01:25:09' 2022-04-10T01:25:25.324 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:25:25.324 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:25:25.324 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:25:25.324 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:25:25.325 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror.mirror_status 2022-04-10T01:25:25.325 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror.mirror_status 2022-04-10T01:25:25.325 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:25:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror.mirror_status 2022-04-10T01:25:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2022-04-10T01:25:25.358 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: failed to query services: (13) Permission denied 2022-04-10T01:25:25.387 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local last_update state 2022-04-10T01:25:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2022-04-10T01:25:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ last_update='2022-04-10 01:25:09' 2022-04-10T01:25:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2022-04-10T01:25:25.392 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ state=up+replaying 2022-04-10T01:25:25.392 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo up+replaying 2022-04-10T01:25:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep up+replaying 2022-04-10T01:25:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:up+replaying 2022-04-10T01:25:25.394 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:25:25.395 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:25:25.395 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ snap_name=snap9 2022-04-10T01:25:25.395 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror test snap9 2022-04-10T01:25:25.395 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:25:25.396 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:25:25.396 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:25:25.396 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap9 2022-04-10T01:25:25.396 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:25:53.277 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror test 2022-04-10T01:25:53.278 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:25:53.278 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:25:53.278 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:25:53.279 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:25:53.279 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:25:53.279 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:25:53.279 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:25:53.279 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2022-04-10T01:25:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:25:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:25:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:25:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:25:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:25:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:25:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:25:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:25:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:25:53.282 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:25:53.282 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:25:54.281 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror test started 2022-04-10T01:25:54.281 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:25:54.281 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:25:54.282 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:25:54.282 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:25:54.282 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:25:54.282 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:25:54.283 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:25:54.283 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:25:54.283 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:25:54.283 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:25:54.284 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2022-04-10T01:25:54.284 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:25:54.284 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:25:54.284 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:25:54.284 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:25:54.285 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:25:54.285 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:25:54.285 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:25:54.285 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:25:54.286 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:25:54.286 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:25:54.286 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:25:54.286 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:25:54.287 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:25:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:25:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:25:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2022-04-10T01:25:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:25:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:25:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:25:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:25:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:25:54.293 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:25:54.293 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:25:54.293 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:25:54.293 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:25:54.293 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:25:54.294 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:25:54.294 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:25:54.294 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:25:54.294 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:25:54.295 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:25:54.295 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:25:54.295 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:25:54.295 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:25:54.296 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:25:54.296 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:25:54.296 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:25:54.296 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:25:54.296 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:25:54.297 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:25:54.297 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:25:54.297 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:25:54.297 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:25:54.297 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:25:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:25:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:25:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:25:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:25:54.299 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:25:54.299 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:25:54.299 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:25:54.299 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:25:54.300 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:25:54.300 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:25:54.304 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:25:54.304 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:25:54.304 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:25:54.305 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2022-04-10T01:25:54.416 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:25:54.417 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:25:54.418 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:25:54.418 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:25:54.418 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:25:54.421 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:25:54.422 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:25:54.422 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:25:54.422 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2022-04-10T01:25:54.422 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:25:54.423 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:25:54.423 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:25:54.423 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:25:54.423 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:25:54.424 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:25:54.424 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:25:54.424 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:25:54.424 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:25:54.424 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:25:54.425 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2022-04-10T01:25:54.425 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:25:54.425 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:25:54.425 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:25:54.425 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:25:54.426 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:25:54.426 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:25:54.426 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:25:54.426 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:25:54.427 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:25:54.427 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:25:54.427 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:25:54.427 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:25:54.428 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-test.status 2022-04-10T01:25:54.428 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-test.status 2022-04-10T01:25:54.428 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:25:54.428 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.status 2022-04-10T01:25:54.428 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-test.status 2022-04-10T01:25:54.429 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-test.status 2022-04-10T01:25:54.429 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:25:54.430 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:25:54.430 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror test 2022-04-10T01:25:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:25:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:25:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:25:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:25:54.432 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:25:54.432 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:25:54.432 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:25:54.432 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2022-04-10T01:25:55.289 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 38 2022-04-10T01:25:55.300 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.rgwz74WspX/cluster2-mirror-test.status 2022-04-10T01:25:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:25:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:25:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:25:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.status 2022-04-10T01:25:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:25:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:25:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:25:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:25:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2022-04-10T01:25:55.303 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:25:55.348 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:25:55.350 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=38 2022-04-10T01:25:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:25:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:25:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:25:55.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:25:55.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:25:55.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:25:55.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:25:55.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:25:55.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:25:55.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:25:55.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:25:55.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:25:55.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2022-04-10T01:25:55.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:25:55.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:25:55.601 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=34 2022-04-10T01:25:55.602 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 38 = 34 2022-04-10T01:25:55.602 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:25:55.602 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:25:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:25:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:25:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:25:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:25:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:25:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:25:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:25:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:25:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:25:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2022-04-10T01:25:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:25:56.055 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:25:56.055 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:25:56.056 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:25:56.056 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:25:56.057 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:25:56.057 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:25:56.058 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 38 = '' 2022-04-10T01:25:56.058 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:25:56.058 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:25:56.859 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:25:56.859 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:25:56.859 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:25:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:25:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:25:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:25:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:25:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:25:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:25:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2022-04-10T01:25:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:25:56.909 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:25:56.910 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:25:56.912 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:25:56.912 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:25:56.912 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:25:56.912 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:25:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 38 = '' 2022-04-10T01:25:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:25:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1.6 2022-04-10T01:25:58.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:25:58.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:25:58.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:25:58.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:25:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:25:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:25:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:25:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:25:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:25:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2022-04-10T01:25:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:25:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:25:58.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=38 2022-04-10T01:25:58.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 38 = 38 2022-04-10T01:25:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:25:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror test snap9 2022-04-10T01:25:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:25:58.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:25:58.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:25:58.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap9 2022-04-10T01:25:58.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:25:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:25:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:25:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:25:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:25:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:25:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:25:59.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/test@snap9 2022-04-10T01:25:59.629 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'test': 2022-04-10T01:25:59.630 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 512 MiB in 128 objects 2022-04-10T01:25:59.630 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:25:59.630 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 18 2022-04-10T01:25:59.631 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 10ae4a893004 2022-04-10T01:25:59.631 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.10ae4a893004 2022-04-10T01:25:59.631 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:25:59.631 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:25:59.632 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:25:59.632 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:25:59.632 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:18:11 2022 2022-04-10T01:25:59.632 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:25:06 2022 2022-04-10T01:25:59.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:18:11 2022 2022-04-10T01:25:59.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:25:59.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:25:59.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:25:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 5ff848f4-a6a0-498d-b9f1-7a041de7de98 2022-04-10T01:25:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:25:59.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:25:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snap8-clean ']' 2022-04-10T01:25:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror test snap8-clean 2022-04-10T01:25:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:25:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:25:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap8-clean 2022-04-10T01:25:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:25:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:25:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:25:59.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:25:59.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:26:00.229 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:26:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:26:01.075 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:26:01.078 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:26:01.335 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:26:01.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:26:01.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror test snap9 2022-04-10T01:26:01.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:26:01.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:26:01.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap9 2022-04-10T01:26:01.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:26:01.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:26:01.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:26:01.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:26:01.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:26:01.895 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.40981747988973505) > weight (0.25) 2022-04-10T01:26:01.895 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.427301478442214) > weight (0.25) 2022-04-10T01:26:01.895 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.5998175109359178) > weight (0.25) 2022-04-10T01:26:01.895 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2022-04-10T01:26:01.896 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2022-04-10T01:26:01.897 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2022-04-10T01:26:01.897 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2022-04-10T01:26:01.897 INFO:teuthology.orchestra.run:waiting for 600 2022-04-10T01:26:01.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi043.stderr:2022-04-10T01:26:01.895+0000 7f1f66872700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 42298) UID: 1000 2022-04-10T01:26:02.091 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:26:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:26:02.097 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi043.stderr:daemon-helper: command failed with exit status 1 2022-04-10T01:26:02.797 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:26:02.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:26:03.056 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:26:03.202 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:26:03.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ clean_snap_name=snap9-clean 2022-04-10T01:26:03.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror test snap9-clean 2022-04-10T01:26:03.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:26:03.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:26:03.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:26:03.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap9-clean 2022-04-10T01:26:03.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9-clean --debug-rbd=20 --debug-journaler=20 2022-04-10T01:26:03.323 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 10` 2022-04-10T01:26:03.324 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror test 2022-04-10T01:26:03.324 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:26:03.325 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:26:03.325 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:26:03.325 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:26:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=13 2022-04-10T01:26:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:26:03.329 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:26:03.329 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:26:03.330 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:26:03.330 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:26:03.330 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2022-04-10T01:26:03.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 73969664 bytes 2022-04-10T01:26:03.841 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 102916096 bytes 2022-04-10T01:26:04.018 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 143309824 bytes 2022-04-10T01:26:04.157 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 143923200 bytes 2022-04-10T01:26:04.393 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 243779584 bytes 2022-04-10T01:26:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 286282752 bytes 2022-04-10T01:26:04.733 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 300693504 bytes 2022-04-10T01:26:04.882 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 302939136 bytes 2022-04-10T01:26:04.950 INFO:tasks.ceph.cluster2.osd.1.smithi043.stderr:2022-04-10T01:26:04.947+0000 7f18011ce700 -1 reset not still connected to 0x55b2c0f45930 2022-04-10T01:26:05.085 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 360314880 bytes 2022-04-10T01:26:05.254 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 397369344 bytes 2022-04-10T01:26:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 404859904 bytes 2022-04-10T01:26:05.562 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 449660928 bytes 2022-04-10T01:26:05.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 450288640 bytes 2022-04-10T01:26:05.908 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 527119360 bytes 2022-04-10T01:26:06.066 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 551156736 bytes 2022-04-10T01:26:06.250 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 600459264 bytes 2022-04-10T01:26:06.469 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 676112384 bytes 2022-04-10T01:26:06.616 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 692553728 bytes 2022-04-10T01:26:06.774 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 721300480 bytes 2022-04-10T01:26:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 780338176 bytes 2022-04-10T01:26:07.124 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 805032960 bytes 2022-04-10T01:26:07.268 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 813688832 bytes 2022-04-10T01:26:07.412 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 833343488 bytes 2022-04-10T01:26:07.568 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 862216192 bytes 2022-04-10T01:26:07.720 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 877491200 bytes 2022-04-10T01:26:07.880 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 902808576 bytes 2022-04-10T01:26:07.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-04-10T01:26:08.000 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2022-04-10T01:26:08.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2022-04-10T01:26:08.000 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2022-04-10T01:26:08.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2022-04-10T01:26:08.005 INFO:tasks.rbd_mirror_thrash:waiting for 105 secs before thrashing 2022-04-10T01:26:08.073 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 961420288 bytes 2022-04-10T01:26:08.265 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 994207744 bytes 2022-04-10T01:26:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1042790400 bytes 2022-04-10T01:26:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1093852160 bytes 2022-04-10T01:26:08.854 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1134558208 bytes 2022-04-10T01:26:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1199149056 bytes 2022-04-10T01:26:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1253994496 bytes 2022-04-10T01:26:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1326346240 bytes 2022-04-10T01:26:09.600 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1354203136 bytes 2022-04-10T01:26:09.843 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1454394368 bytes 2022-04-10T01:26:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1500792832 bytes 2022-04-10T01:26:10.213 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1530293248 bytes 2022-04-10T01:26:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1538691072 bytes 2022-04-10T01:26:10.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1618264064 bytes 2022-04-10T01:26:10.792 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2022-04-10T01:26:10.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2022-04-10T01:26:10.793 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2022-04-10T01:26:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1683728384 bytes 2022-04-10T01:26:10.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi043.stderr:2022-04-10T01:26:10.791+0000 7f1616f18700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 39450) UID: 1000 2022-04-10T01:26:10.990 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 1733539840 bytes 2022-04-10T01:26:10.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi043.stderr:daemon-helper: command failed with exit status 1 2022-04-10T01:26:11.194 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 1807975424 bytes 2022-04-10T01:26:11.350 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 1824915456 bytes 2022-04-10T01:26:11.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 1856584704 bytes 2022-04-10T01:26:11.729 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 1927571456 bytes 2022-04-10T01:26:11.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 1939576832 bytes 2022-04-10T01:26:12.065 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 1977281536 bytes 2022-04-10T01:26:12.225 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 2006658048 bytes 2022-04-10T01:26:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 2017491968 bytes 2022-04-10T01:26:12.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 2096881664 bytes 2022-04-10T01:26:12.826 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 2168569856 bytes 2022-04-10T01:26:12.974 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 2186885120 bytes 2022-04-10T01:26:13.123 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5400 IOs, 2201939968 bytes 2022-04-10T01:26:13.315 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5500 IOs, 2250734592 bytes 2022-04-10T01:26:13.470 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5600 IOs, 2280520704 bytes 2022-04-10T01:26:13.648 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5700 IOs, 2318559232 bytes 2022-04-10T01:26:13.833 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5800 IOs, 2355235840 bytes 2022-04-10T01:26:14.015 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5900 IOs, 2395055104 bytes 2022-04-10T01:26:14.221 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6000 IOs, 2455334912 bytes 2022-04-10T01:26:14.384 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6100 IOs, 2488086528 bytes 2022-04-10T01:26:14.530 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6200 IOs, 2488689664 bytes 2022-04-10T01:26:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6300 IOs, 2520696832 bytes 2022-04-10T01:26:14.883 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6400 IOs, 2551137280 bytes 2022-04-10T01:26:15.075 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6500 IOs, 2609863680 bytes 2022-04-10T01:26:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6600 IOs, 2680596480 bytes 2022-04-10T01:26:15.458 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6700 IOs, 2732049408 bytes 2022-04-10T01:26:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6800 IOs, 2769058816 bytes 2022-04-10T01:26:15.809 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6900 IOs, 2787540992 bytes 2022-04-10T01:26:16.026 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7000 IOs, 2856813568 bytes 2022-04-10T01:26:16.188 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7100 IOs, 2878377984 bytes 2022-04-10T01:26:16.332 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:26:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:26:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:26:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:26:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2022-04-10T01:26:16.334 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:26:16.334 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:26:16.334 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:26:16.334 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state_pattern=up+replaying 2022-04-10T01:26:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local description_pattern= 2022-04-10T01:26:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local service_pattern= 2022-04-10T01:26:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:26:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:26:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:26:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:26:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:26:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:26:16.794 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2022-04-10T01:26:16.795 INFO:teuthology.orchestra.run:waiting for 600 2022-04-10T01:26:16.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-04-10T01:26:16.795 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2022-04-10T01:26:16.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2022-04-10T01:26:16.796 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2022-04-10T01:26:16.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2022-04-10T01:26:16.799 INFO:tasks.rbd_mirror_thrash:waiting for 71 secs before thrashing 2022-04-10T01:26:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2022-04-10T01:26:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:26:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:26:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:26:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state_pattern=up+replaying 2022-04-10T01:26:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local description_pattern= 2022-04-10T01:26:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local service_pattern= 2022-04-10T01:26:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:26:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:26:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:26:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:26:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-test.mirror_status 2022-04-10T01:26:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-test.mirror_status 2022-04-10T01:26:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:26:17.338 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:26:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ CEPH_ARGS= 2022-04-10T01:26:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2022-04-10T01:26:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ tee /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:26:17.382 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:test: 2022-04-10T01:26:17.382 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: global_id: 5ff848f4-a6a0-498d-b9f1-7a041de7de98 2022-04-10T01:26:17.382 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: state: down+unknown 2022-04-10T01:26:17.383 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: description: status not found 2022-04-10T01:26:17.383 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: last_update: 2022-04-10T01:26:17.383 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: peer_sites: 2022-04-10T01:26:17.383 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: name: cluster2 2022-04-10T01:26:17.384 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: state: up+stopped 2022-04-10T01:26:17.384 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: description: local image is primary 2022-04-10T01:26:17.384 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: last_update: 2022-04-10 01:26:09 2022-04-10T01:26:17.388 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:26:17.390 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 1 2022-04-10T01:26:17.390 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:26:17.390 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 2 2022-04-10T01:26:19.392 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2022-04-10T01:26:19.392 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:26:19.393 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:26:19.393 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:26:19.393 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state_pattern=up+replaying 2022-04-10T01:26:19.393 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local description_pattern= 2022-04-10T01:26:19.394 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local service_pattern= 2022-04-10T01:26:19.394 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:26:19.394 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:26:19.394 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:26:19.394 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:26:19.395 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-test.mirror_status 2022-04-10T01:26:19.395 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-test.mirror_status 2022-04-10T01:26:19.395 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:26:19.396 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:26:19.396 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ CEPH_ARGS= 2022-04-10T01:26:19.397 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2022-04-10T01:26:19.397 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ tee /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:26:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:test: 2022-04-10T01:26:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: global_id: 5ff848f4-a6a0-498d-b9f1-7a041de7de98 2022-04-10T01:26:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: state: down+unknown 2022-04-10T01:26:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: description: status not found 2022-04-10T01:26:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: last_update: 2022-04-10T01:26:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: peer_sites: 2022-04-10T01:26:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: name: cluster2 2022-04-10T01:26:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: state: up+stopped 2022-04-10T01:26:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: description: local image is primary 2022-04-10T01:26:19.445 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: last_update: 2022-04-10 01:26:09 2022-04-10T01:26:19.448 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:26:19.449 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 1 2022-04-10T01:26:19.449 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:26:19.450 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 4 2022-04-10T01:26:23.452 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2022-04-10T01:26:23.452 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:26:23.453 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:26:23.453 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:26:23.453 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state_pattern=up+replaying 2022-04-10T01:26:23.453 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local description_pattern= 2022-04-10T01:26:23.453 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local service_pattern= 2022-04-10T01:26:23.454 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:26:23.454 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:26:23.455 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:26:23.455 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:26:23.455 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-test.mirror_status 2022-04-10T01:26:23.455 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-test.mirror_status 2022-04-10T01:26:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:26:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:26:23.457 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ CEPH_ARGS= 2022-04-10T01:26:23.457 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2022-04-10T01:26:23.457 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ tee /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:26:23.502 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:test: 2022-04-10T01:26:23.502 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: global_id: 5ff848f4-a6a0-498d-b9f1-7a041de7de98 2022-04-10T01:26:23.502 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: state: down+unknown 2022-04-10T01:26:23.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: description: status not found 2022-04-10T01:26:23.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: last_update: 2022-04-10T01:26:23.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: peer_sites: 2022-04-10T01:26:23.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: name: cluster2 2022-04-10T01:26:23.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: state: up+stopped 2022-04-10T01:26:23.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: description: local image is primary 2022-04-10T01:26:23.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: last_update: 2022-04-10 01:26:09 2022-04-10T01:26:23.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:26:23.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 1 2022-04-10T01:26:23.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:26:23.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 8 2022-04-10T01:26:31.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2022-04-10T01:26:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:26:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:26:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:26:31.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state_pattern=up+replaying 2022-04-10T01:26:31.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local description_pattern= 2022-04-10T01:26:31.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local service_pattern= 2022-04-10T01:26:31.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:26:31.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:26:31.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:26:31.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:26:31.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-test.mirror_status 2022-04-10T01:26:31.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-test.mirror_status 2022-04-10T01:26:31.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:26:31.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:26:31.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ CEPH_ARGS= 2022-04-10T01:26:31.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2022-04-10T01:26:31.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ tee /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:26:31.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:test: 2022-04-10T01:26:31.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: global_id: 5ff848f4-a6a0-498d-b9f1-7a041de7de98 2022-04-10T01:26:31.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: state: down+unknown 2022-04-10T01:26:31.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: description: status not found 2022-04-10T01:26:31.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: last_update: 2022-04-10T01:26:31.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: peer_sites: 2022-04-10T01:26:31.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: name: cluster2 2022-04-10T01:26:31.562 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: state: up+stopped 2022-04-10T01:26:31.562 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: description: local image is primary 2022-04-10T01:26:31.562 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: last_update: 2022-04-10 01:26:09 2022-04-10T01:26:31.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:26:31.566 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 1 2022-04-10T01:26:31.567 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:26:31.567 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 8 2022-04-10T01:26:39.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2022-04-10T01:26:39.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:26:39.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:26:39.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:26:39.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state_pattern=up+replaying 2022-04-10T01:26:39.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local description_pattern= 2022-04-10T01:26:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local service_pattern= 2022-04-10T01:26:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:26:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:26:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:26:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:26:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-test.mirror_status 2022-04-10T01:26:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-test.mirror_status 2022-04-10T01:26:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:26:39.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:26:39.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ CEPH_ARGS= 2022-04-10T01:26:39.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2022-04-10T01:26:39.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ tee /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:26:39.617 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:test: 2022-04-10T01:26:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: global_id: 5ff848f4-a6a0-498d-b9f1-7a041de7de98 2022-04-10T01:26:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: state: down+unknown 2022-04-10T01:26:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: description: status not found 2022-04-10T01:26:39.619 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: last_update: 2022-04-10T01:26:39.619 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: peer_sites: 2022-04-10T01:26:39.619 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: name: cluster2 2022-04-10T01:26:39.619 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: state: up+stopped 2022-04-10T01:26:39.619 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: description: local image is primary 2022-04-10T01:26:39.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: last_update: 2022-04-10 01:26:39 2022-04-10T01:26:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:26:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 1 2022-04-10T01:26:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:26:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 8 2022-04-10T01:26:47.626 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2022-04-10T01:26:47.626 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:26:47.627 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:26:47.627 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:26:47.627 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state_pattern=up+replaying 2022-04-10T01:26:47.627 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local description_pattern= 2022-04-10T01:26:47.628 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local service_pattern= 2022-04-10T01:26:47.628 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:26:47.628 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:26:47.628 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:26:47.628 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:26:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-test.mirror_status 2022-04-10T01:26:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-test.mirror_status 2022-04-10T01:26:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:26:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:26:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ CEPH_ARGS= 2022-04-10T01:26:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2022-04-10T01:26:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ tee /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:26:47.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:test: 2022-04-10T01:26:47.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: global_id: 5ff848f4-a6a0-498d-b9f1-7a041de7de98 2022-04-10T01:26:47.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: state: up+replaying 2022-04-10T01:26:47.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1649553955,"remote_snapshot_timestamp":1649553955,"replay_state":"idle"} 2022-04-10T01:26:47.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: service: mirror.3 on smithi043 2022-04-10T01:26:47.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: last_update: 2022-04-10 01:26:43 2022-04-10T01:26:47.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: peer_sites: 2022-04-10T01:26:47.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: name: cluster2 2022-04-10T01:26:47.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: state: up+stopped 2022-04-10T01:26:47.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: description: local image is primary 2022-04-10T01:26:47.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: last_update: 2022-04-10 01:26:39 2022-04-10T01:26:47.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:26:47.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: state: up+replaying 2022-04-10T01:26:47.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep '^ description: .*' /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:26:47.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1649553955,"remote_snapshot_timestamp":1649553955,"replay_state":"idle"} 2022-04-10T01:26:47.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:26:47.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo up+replaying 2022-04-10T01:26:47.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep '^up+' 2022-04-10T01:26:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:up+replaying 2022-04-10T01:26:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:26:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: service: mirror.3 on smithi043 2022-04-10T01:26:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.rgwz74WspX/cluster1-mirror-test.mirror_status 2022-04-10T01:26:47.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local 'last_update=2022-04-10 01:26:43' 2022-04-10T01:26:47.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2022-04-10 01:26:43' 2022-04-10T01:26:47.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:26:47.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:26:47.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:26:47.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state_pattern=up+replaying 2022-04-10T01:26:47.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local 'prev_last_update=2022-04-10 01:26:43' 2022-04-10T01:26:47.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:26:47.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:26:47.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:26:47.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:26:47.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror.mirror_status 2022-04-10T01:26:47.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror.mirror_status 2022-04-10T01:26:47.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:26:47.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror.mirror_status 2022-04-10T01:26:47.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2022-04-10T01:26:47.725 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: failed to query services: (13) Permission denied 2022-04-10T01:26:47.752 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local last_update state 2022-04-10T01:26:47.753 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2022-04-10T01:26:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ last_update='2022-04-10 01:26:43' 2022-04-10T01:26:47.755 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2022-04-10T01:26:47.757 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ state=up+replaying 2022-04-10T01:26:47.757 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo up+replaying 2022-04-10T01:26:47.757 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep up+replaying 2022-04-10T01:26:47.758 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:up+replaying 2022-04-10T01:26:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:26:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:26:47.761 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ snap_name=snap10 2022-04-10T01:26:47.761 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror test snap10 2022-04-10T01:26:47.761 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:26:47.761 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:26:47.762 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:26:47.762 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap10 2022-04-10T01:26:47.762 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:26:49.271 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror test 2022-04-10T01:26:49.271 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:26:49.271 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:26:49.272 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:26:49.272 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:26:49.272 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:26:49.272 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:26:49.273 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:26:49.273 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2022-04-10T01:26:49.273 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:26:49.273 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:26:49.274 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:26:49.274 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:26:49.274 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:26:49.274 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:26:49.275 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:26:49.275 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:26:49.275 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:26:49.276 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:26:49.276 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:26:50.273 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror test started 2022-04-10T01:26:50.274 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:26:50.274 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:26:50.274 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:26:50.275 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:26:50.275 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:26:50.275 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:26:50.275 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:26:50.275 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:26:50.276 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:26:50.276 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:26:50.276 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2022-04-10T01:26:50.276 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:26:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:26:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:26:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:26:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:26:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:26:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:26:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:26:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:26:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:26:50.279 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:26:50.279 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:26:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:26:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:26:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:26:50.284 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2022-04-10T01:26:50.284 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:26:50.284 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:26:50.284 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:26:50.284 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:26:50.285 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:26:50.285 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:26:50.285 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:26:50.285 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:26:50.286 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:26:50.286 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:26:50.286 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:26:50.286 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:26:50.287 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:26:50.287 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:26:50.287 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:26:50.287 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:26:50.287 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:26:50.288 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:26:50.288 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:26:50.288 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:26:50.288 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:26:50.289 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:26:50.289 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:26:50.289 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:26:50.289 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:26:50.289 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:26:50.290 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:26:50.290 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:26:50.290 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:26:50.290 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:26:50.291 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:26:50.291 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:26:50.291 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:26:50.291 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:26:50.292 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:26:50.292 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:26:50.292 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:26:50.292 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:26:50.296 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:26:50.296 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:26:50.296 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:26:50.296 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2022-04-10T01:26:50.397 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/test'] 2022-04-10T01:26:50.398 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:26:50.398 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:26:50.403 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:26:50.403 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2022-04-10T01:26:50.403 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:1 2022-04-10T01:26:50.404 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:26:50.404 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:26:50.404 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:1 cluster instance 2022-04-10T01:26:50.404 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:1 2022-04-10T01:26:50.404 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:26:50.405 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:26:50.405 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:26:50.405 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=1 2022-04-10T01:26:50.405 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 1 = cluster1:1 ']' 2022-04-10T01:26:50.406 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:26:50.406 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:26:50.406 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=1 2022-04-10T01:26:50.406 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:26:50.407 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:26:50.407 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:26:50.407 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:26:50.407 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:26:50.408 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:1 cluster1 2022-04-10T01:26:50.408 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:1 2022-04-10T01:26:50.408 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:26:50.408 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:26:50.408 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2022-04-10T01:26:50.409 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:1 2022-04-10T01:26:50.409 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:26:50.409 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:26:50.409 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:26:50.410 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:26:50.410 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 1 = cluster1:1 ']' 2022-04-10T01:26:50.410 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:26:50.410 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:26:50.411 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=1 2022-04-10T01:26:50.411 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=1 2022-04-10T01:26:50.411 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:26:50.411 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:26:50.412 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:26:50.412 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:26:50.412 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2022-04-10T01:26:50.416 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:26:50.417 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:26:50.417 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:26:50.417 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2022-04-10T01:26:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/test'] 2022-04-10T01:26:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:26:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:26:50.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:26:50.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2022-04-10T01:26:50.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:2 2022-04-10T01:26:50.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:26:50.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:26:50.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:2 cluster instance 2022-04-10T01:26:50.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:2 2022-04-10T01:26:50.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:26:50.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:26:50.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:26:50.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=2 2022-04-10T01:26:50.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 2 = cluster1:2 ']' 2022-04-10T01:26:50.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:26:50.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:26:50.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=2 2022-04-10T01:26:50.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:26:50.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:26:50.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:26:50.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:26:50.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:26:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:2 cluster1 2022-04-10T01:26:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:2 2022-04-10T01:26:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:26:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:26:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2022-04-10T01:26:50.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:2 2022-04-10T01:26:50.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:26:50.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:26:50.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:26:50.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:26:50.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 2 = cluster1:2 ']' 2022-04-10T01:26:50.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:26:50.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:26:50.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=2 2022-04-10T01:26:50.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=2 2022-04-10T01:26:50.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:26:50.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:26:50.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:26:50.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:26:50.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2022-04-10T01:26:50.529 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:26:50.529 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:26:50.529 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:26:50.530 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2022-04-10T01:26:50.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/test'] 2022-04-10T01:26:50.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:26:50.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:26:50.632 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:26:50.632 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2022-04-10T01:26:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:3 2022-04-10T01:26:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:26:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:26:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:3 cluster instance 2022-04-10T01:26:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:3 2022-04-10T01:26:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:26:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:26:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:26:50.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=3 2022-04-10T01:26:50.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 3 = cluster1:3 ']' 2022-04-10T01:26:50.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:26:50.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:26:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=3 2022-04-10T01:26:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=3 2022-04-10T01:26:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:26:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:26:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:26:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:26:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:3 cluster1 2022-04-10T01:26:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:3 2022-04-10T01:26:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:26:50.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:26:50.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2022-04-10T01:26:50.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:3 2022-04-10T01:26:50.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:26:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:26:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:26:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=3 2022-04-10T01:26:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 3 = cluster1:3 ']' 2022-04-10T01:26:50.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:26:50.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:26:50.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=3 2022-04-10T01:26:50.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=3 2022-04-10T01:26:50.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:26:50.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:26:50.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:26:50.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:26:50.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2022-04-10T01:26:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:26:50.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:26:50.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:26:50.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2022-04-10T01:26:50.745 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:26:50.746 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:26:50.747 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:26:50.747 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:26:50.747 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:26:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:26:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:26:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:26:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2022-04-10T01:26:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:26:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:26:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:26:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:26:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:26:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:26:50.755 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:26:50.755 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:26:50.755 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:26:50.755 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:26:50.755 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2022-04-10T01:26:50.756 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:26:50.756 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:26:50.756 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:26:50.756 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:26:50.757 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:26:50.757 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:26:50.757 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:26:50.757 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:26:50.758 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:26:50.758 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:26:50.758 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:26:50.758 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:26:50.759 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-test.status 2022-04-10T01:26:50.759 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-test.status 2022-04-10T01:26:50.759 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:26:50.759 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.status 2022-04-10T01:26:50.759 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-test.status 2022-04-10T01:26:50.760 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-test.status 2022-04-10T01:26:50.760 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:26:50.760 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:26:50.760 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror test 2022-04-10T01:26:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:26:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:26:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:26:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:26:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:26:50.762 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:26:50.762 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:26:50.762 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2022-04-10T01:26:51.277 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 42 2022-04-10T01:26:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.rgwz74WspX/cluster2-mirror-test.status 2022-04-10T01:26:51.291 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:26:51.291 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:26:51.291 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:26:51.291 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.status 2022-04-10T01:26:51.292 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:26:51.292 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:26:51.292 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:26:51.292 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:26:51.293 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2022-04-10T01:26:51.293 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:26:51.334 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:26:51.335 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=42 2022-04-10T01:26:51.336 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:26:51.336 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:26:51.336 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:26:51.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:26:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:26:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:26:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:26:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:26:51.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:26:51.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:26:51.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:26:51.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:26:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2022-04-10T01:26:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:26:51.587 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:26:51.589 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=38 2022-04-10T01:26:51.589 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 42 = 38 2022-04-10T01:26:51.589 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:26:51.589 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:26:51.991 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:26:51.991 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:26:51.992 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:26:51.992 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:26:51.992 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:26:51.992 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:26:51.992 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:26:51.993 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:26:51.993 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:26:51.993 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2022-04-10T01:26:51.993 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:26:52.246 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:26:52.247 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:26:52.248 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:26:52.248 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:26:52.249 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:26:52.249 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:26:52.250 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 42 = '' 2022-04-10T01:26:52.250 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:26:52.250 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:26:53.051 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:26:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:26:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:26:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:26:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:26:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:26:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:26:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:26:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:26:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2022-04-10T01:26:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:26:53.105 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:26:53.106 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:26:53.108 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:26:53.108 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:26:53.108 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:26:53.108 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:26:53.109 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 42 = '' 2022-04-10T01:26:53.109 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:26:53.109 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1.6 2022-04-10T01:26:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:26:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:26:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:26:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:26:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:26:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:26:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:26:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:26:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:26:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2022-04-10T01:26:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:26:54.755 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:26:54.757 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=42 2022-04-10T01:26:54.758 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 42 = 42 2022-04-10T01:26:54.758 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:26:54.758 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror test snap10 2022-04-10T01:26:54.759 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:26:54.759 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:26:54.759 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:26:54.759 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap10 2022-04-10T01:26:54.759 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:26:54.760 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:26:54.760 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:26:54.760 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:26:54.760 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:26:54.761 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:26:54.761 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:26:55.758 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/test@snap10 2022-04-10T01:26:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'test': 2022-04-10T01:26:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 512 MiB in 128 objects 2022-04-10T01:26:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:26:55.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 20 2022-04-10T01:26:55.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 10ae4a893004 2022-04-10T01:26:55.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.10ae4a893004 2022-04-10T01:26:55.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:26:55.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:26:55.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:26:55.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:26:55.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:18:11 2022 2022-04-10T01:26:55.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:25:06 2022 2022-04-10T01:26:55.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:18:11 2022 2022-04-10T01:26:55.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:26:55.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:26:55.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:26:55.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 5ff848f4-a6a0-498d-b9f1-7a041de7de98 2022-04-10T01:26:55.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:26:55.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:26:55.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snap9-clean ']' 2022-04-10T01:26:55.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror test snap9-clean 2022-04-10T01:26:55.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:26:55.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:26:55.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap9-clean 2022-04-10T01:26:55.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:26:55.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:26:55.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:26:55.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:26:55.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:26:56.394 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:26:56.397 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:26:57.208 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:26:57.212 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:26:57.470 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:26:57.615 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:26:57.615 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror test snap10 2022-04-10T01:26:57.615 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:26:57.615 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:26:57.616 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap10 2022-04-10T01:26:57.616 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:26:57.616 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:26:57.616 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:26:57.617 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:26:57.617 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:26:58.226 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:26:58.229 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:26:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:26:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:26:59.272 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:26:59.417 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:26:59.417 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ clean_snap_name=snap10-clean 2022-04-10T01:26:59.418 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror test snap10-clean 2022-04-10T01:26:59.418 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:26:59.418 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:26:59.418 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:26:59.419 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap10-clean 2022-04-10T01:26:59.419 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10-clean --debug-rbd=20 --debug-journaler=20 2022-04-10T01:27:00.281 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror test 2022-04-10T01:27:00.282 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:27:00.282 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:27:00.282 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:27:00.283 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:27:00.283 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:27:00.283 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:27:00.283 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:00.284 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2022-04-10T01:27:00.284 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:27:00.284 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:27:00.284 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:27:00.284 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:27:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:27:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:27:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:27:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:27:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:27:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:27:01.284 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror test started 2022-04-10T01:27:01.285 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:27:01.285 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:27:01.285 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:27:01.285 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:27:01.286 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:27:01.286 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:27:01.286 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:27:01.286 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:27:01.287 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:27:01.287 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:01.287 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2022-04-10T01:27:01.287 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:27:01.288 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:27:01.288 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:27:01.288 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:27:01.288 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:27:01.289 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:27:01.289 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:27:01.289 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:27:01.289 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:27:01.290 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:27:01.290 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:27:01.290 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:27:01.290 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:27:01.290 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:27:01.294 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:27:01.295 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2022-04-10T01:27:01.295 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:27:01.295 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:27:01.295 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:27:01.296 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:27:01.296 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:27:01.296 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:27:01.296 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:27:01.296 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:27:01.297 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:27:01.297 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:27:01.297 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:27:01.297 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:27:01.297 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:27:01.298 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:27:01.298 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:27:01.298 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:27:01.298 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:27:01.299 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:27:01.299 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:27:01.299 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:27:01.299 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:27:01.300 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:27:01.300 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:27:01.300 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:27:01.300 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:27:01.300 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:27:01.301 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:27:01.301 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:27:01.301 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:27:01.301 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:27:01.301 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:27:01.302 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:27:01.302 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:27:01.302 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:27:01.302 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:27:01.303 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:27:01.303 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:27:01.303 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:27:01.306 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:27:01.306 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:27:01.306 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:27:01.307 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2022-04-10T01:27:01.407 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/test'] 2022-04-10T01:27:01.408 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:27:01.408 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:27:01.412 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:27:01.412 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2022-04-10T01:27:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:1 2022-04-10T01:27:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:27:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:27:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:1 cluster instance 2022-04-10T01:27:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:1 2022-04-10T01:27:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:27:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:27:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:27:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=1 2022-04-10T01:27:01.415 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 1 = cluster1:1 ']' 2022-04-10T01:27:01.415 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:27:01.415 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:27:01.416 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=1 2022-04-10T01:27:01.416 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:27:01.416 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:27:01.416 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:27:01.416 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:27:01.417 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:27:01.417 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:1 cluster1 2022-04-10T01:27:01.417 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:1 2022-04-10T01:27:01.417 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:27:01.418 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:27:01.418 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2022-04-10T01:27:01.418 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:1 2022-04-10T01:27:01.418 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:27:01.418 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:27:01.419 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:27:01.419 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:27:01.419 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 1 = cluster1:1 ']' 2022-04-10T01:27:01.419 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:27:01.420 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:27:01.420 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=1 2022-04-10T01:27:01.420 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=1 2022-04-10T01:27:01.421 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:27:01.421 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:27:01.421 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:27:01.421 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:27:01.421 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2022-04-10T01:27:01.426 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:27:01.426 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:27:01.427 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:27:01.427 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2022-04-10T01:27:01.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/test'] 2022-04-10T01:27:01.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:27:01.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:27:01.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:27:01.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2022-04-10T01:27:01.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:2 2022-04-10T01:27:01.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:27:01.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:27:01.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:2 cluster instance 2022-04-10T01:27:01.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:2 2022-04-10T01:27:01.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:27:01.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:27:01.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:27:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=2 2022-04-10T01:27:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 2 = cluster1:2 ']' 2022-04-10T01:27:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:27:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:27:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=2 2022-04-10T01:27:01.529 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:27:01.529 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:27:01.529 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:27:01.529 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:27:01.530 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:27:01.530 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:2 cluster1 2022-04-10T01:27:01.530 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:2 2022-04-10T01:27:01.530 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:27:01.530 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:27:01.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2022-04-10T01:27:01.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:2 2022-04-10T01:27:01.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:27:01.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:27:01.532 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:27:01.532 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:27:01.532 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 2 = cluster1:2 ']' 2022-04-10T01:27:01.532 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:27:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:27:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=2 2022-04-10T01:27:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=2 2022-04-10T01:27:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:27:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:27:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:27:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:27:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2022-04-10T01:27:01.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:27:01.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:27:01.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:27:01.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2022-04-10T01:27:01.632 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/test'] 2022-04-10T01:27:01.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:27:01.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:27:01.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:27:01.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2022-04-10T01:27:01.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:3 2022-04-10T01:27:01.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:27:01.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:27:01.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:3 cluster instance 2022-04-10T01:27:01.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:3 2022-04-10T01:27:01.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:27:01.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:27:01.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:27:01.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=3 2022-04-10T01:27:01.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 3 = cluster1:3 ']' 2022-04-10T01:27:01.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:27:01.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:27:01.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=3 2022-04-10T01:27:01.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=3 2022-04-10T01:27:01.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:27:01.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:27:01.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:27:01.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:27:01.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:3 cluster1 2022-04-10T01:27:01.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:3 2022-04-10T01:27:01.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:27:01.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:27:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2022-04-10T01:27:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:3 2022-04-10T01:27:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:27:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:27:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:27:01.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=3 2022-04-10T01:27:01.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 3 = cluster1:3 ']' 2022-04-10T01:27:01.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:27:01.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:27:01.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=3 2022-04-10T01:27:01.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=3 2022-04-10T01:27:01.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:27:01.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:27:01.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:27:01.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:27:01.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2022-04-10T01:27:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:27:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:27:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:27:01.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2022-04-10T01:27:01.754 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:27:01.755 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:27:01.755 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:27:01.755 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:27:01.756 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:27:01.760 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:27:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:27:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2022-04-10T01:27:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:27:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:27:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:27:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:27:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:27:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:27:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:27:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:27:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:27:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2022-04-10T01:27:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:27:01.765 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:27:01.765 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:27:01.765 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:27:01.765 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:27:01.766 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:27:01.766 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:27:01.766 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:01.766 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:27:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:27:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:27:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-test.status 2022-04-10T01:27:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-test.status 2022-04-10T01:27:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:27:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.status 2022-04-10T01:27:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-test.status 2022-04-10T01:27:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-test.status 2022-04-10T01:27:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:27:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:27:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror test 2022-04-10T01:27:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:27:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:27:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:27:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:27:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:27:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:27:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2022-04-10T01:27:02.289 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 45 2022-04-10T01:27:02.303 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.rgwz74WspX/cluster2-mirror-test.status 2022-04-10T01:27:02.303 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:27:02.303 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:27:02.303 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:27:02.304 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.status 2022-04-10T01:27:02.304 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:27:02.304 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:27:02.304 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:27:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2022-04-10T01:27:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:27:02.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:27:02.347 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=45 2022-04-10T01:27:02.348 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:27:02.348 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:27:02.348 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:27:02.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:27:02.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:27:02.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:27:02.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:27:02.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:27:02.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:27:02.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:27:02.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:27:02.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:02.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2022-04-10T01:27:02.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:27:02.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:27:02.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:27:02.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:27:02.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:27:02.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:27:02.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:27:02.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 45 = '' 2022-04-10T01:27:02.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:27:02.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:27:02.998 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:27:02.999 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:27:02.999 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:27:03.000 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:27:03.000 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:27:03.000 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:27:03.000 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:27:03.000 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:27:03.001 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:03.001 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2022-04-10T01:27:03.001 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:27:03.047 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:27:03.047 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:27:03.048 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:27:03.048 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:27:03.049 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:27:03.049 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:27:03.049 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 45 = '' 2022-04-10T01:27:03.050 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:27:03.050 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:27:03.851 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:27:03.851 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:27:03.852 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:27:03.852 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:27:03.852 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.status 2022-04-10T01:27:03.852 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:27:03.853 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:27:03.853 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:27:03.853 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:03.853 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2022-04-10T01:27:03.853 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:27:03.905 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:27:03.908 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=45 2022-04-10T01:27:03.908 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 45 = 45 2022-04-10T01:27:03.909 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:03.909 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror test snap10-clean 2022-04-10T01:27:03.909 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:27:03.909 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:27:03.910 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:27:03.910 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap10-clean 2022-04-10T01:27:03.910 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:27:03.910 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:27:03.911 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:27:03.911 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:27:03.911 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:03.911 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:27:03.911 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:27:04.908 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/test@snap10-clean 2022-04-10T01:27:04.954 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'test': 2022-04-10T01:27:04.955 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 512 MiB in 128 objects 2022-04-10T01:27:04.955 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:27:04.955 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 21 2022-04-10T01:27:04.955 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 10ae4a893004 2022-04-10T01:27:04.956 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.10ae4a893004 2022-04-10T01:27:04.956 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:27:04.956 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:27:04.956 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:27:04.957 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:27:04.957 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:18:11 2022 2022-04-10T01:27:04.957 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:26:56 2022 2022-04-10T01:27:04.957 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:18:11 2022 2022-04-10T01:27:04.957 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:27:04.958 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:27:04.958 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:27:04.958 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 5ff848f4-a6a0-498d-b9f1-7a041de7de98 2022-04-10T01:27:04.958 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:27:04.960 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:04.960 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ seq 1 10 2022-04-10T01:27:04.961 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 10` 2022-04-10T01:27:04.961 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ snap_name=snap1 2022-04-10T01:27:04.961 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror test snap1 2022-04-10T01:27:04.961 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:27:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:27:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap1 2022-04-10T01:27:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:27:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:27:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:27:05.556 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:27:05.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:06.362 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:27:06.365 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:06.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:06.766 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ snap_name=snap1-clean 2022-04-10T01:27:06.766 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror test snap1-clean 2022-04-10T01:27:06.766 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:27:06.766 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:27:06.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap1-clean 2022-04-10T01:27:06.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:27:06.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:27:06.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:06.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:06.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:27:07.358 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:27:07.361 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:08.065 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:27:08.068 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:08.324 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:08.470 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:08.470 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 10` 2022-04-10T01:27:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ snap_name=snap2 2022-04-10T01:27:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror test snap2 2022-04-10T01:27:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:27:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:27:08.472 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap2 2022-04-10T01:27:08.472 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:27:08.472 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:27:08.472 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:08.473 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:08.473 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:27:09.073 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:27:09.077 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:09.830 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:27:09.833 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:10.088 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:10.234 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:10.234 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ snap_name=snap2-clean 2022-04-10T01:27:10.234 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror test snap2-clean 2022-04-10T01:27:10.235 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:27:10.235 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:27:10.235 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap2-clean 2022-04-10T01:27:10.235 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:27:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:27:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:27:10.838 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:27:10.842 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:11.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:27:11.567 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:11.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:11.969 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:11.969 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 10` 2022-04-10T01:27:11.969 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ snap_name=snap3 2022-04-10T01:27:11.970 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror test snap3 2022-04-10T01:27:11.970 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:27:11.970 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:27:11.970 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap3 2022-04-10T01:27:11.971 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:27:11.971 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:27:11.971 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:11.971 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:11.971 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:27:12.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:27:12.584 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:27:13.305 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:13.562 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:13.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ snap_name=snap3-clean 2022-04-10T01:27:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror test snap3-clean 2022-04-10T01:27:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:27:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:27:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap3-clean 2022-04-10T01:27:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:27:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:27:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:27:14.317 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:27:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:15.125 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:27:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:15.387 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:15.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:15.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 10` 2022-04-10T01:27:15.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ snap_name=snap4 2022-04-10T01:27:15.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror test snap4 2022-04-10T01:27:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:27:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:27:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap4 2022-04-10T01:27:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:27:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:27:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:27:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:27:16.133 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:16.886 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:27:16.889 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:17.145 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:17.293 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:17.293 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ snap_name=snap4-clean 2022-04-10T01:27:17.293 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror test snap4-clean 2022-04-10T01:27:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:27:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:27:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap4-clean 2022-04-10T01:27:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:27:17.295 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:27:17.295 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:17.295 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:17.295 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:27:17.882 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:27:17.886 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:18.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:27:18.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:18.968 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:19.118 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:19.118 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 10` 2022-04-10T01:27:19.119 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ snap_name=snap5 2022-04-10T01:27:19.119 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror test snap5 2022-04-10T01:27:19.119 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:27:19.119 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:27:19.120 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap5 2022-04-10T01:27:19.120 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:27:19.120 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:27:19.121 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:19.121 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:19.121 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:27:19.702 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:27:19.705 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:20.450 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:27:20.453 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:20.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:20.858 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:20.859 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ snap_name=snap5-clean 2022-04-10T01:27:20.859 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror test snap5-clean 2022-04-10T01:27:20.859 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:27:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:27:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap5-clean 2022-04-10T01:27:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:27:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:27:20.861 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:20.861 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:20.861 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:27:21.467 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:27:21.471 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:22.163 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:27:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:22.423 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:22.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:22.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 10` 2022-04-10T01:27:22.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ snap_name=snap6 2022-04-10T01:27:22.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror test snap6 2022-04-10T01:27:22.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:27:22.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:27:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap6 2022-04-10T01:27:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:27:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:27:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:22.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:22.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:27:23.163 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:27:23.166 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:27:23.995 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:24.251 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:24.399 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:24.399 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ snap_name=snap6-clean 2022-04-10T01:27:24.400 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror test snap6-clean 2022-04-10T01:27:24.400 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:27:24.400 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:27:24.401 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap6-clean 2022-04-10T01:27:24.401 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:27:24.401 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:27:24.402 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:24.402 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:24.403 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:27:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:27:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:25.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:27:25.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:25.942 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:26.090 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:26.090 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 10` 2022-04-10T01:27:26.090 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ snap_name=snap7 2022-04-10T01:27:26.090 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror test snap7 2022-04-10T01:27:26.091 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:27:26.091 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:27:26.091 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap7 2022-04-10T01:27:26.091 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:27:26.092 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:27:26.092 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:26.092 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:26.092 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:27:26.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:27:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:27:27.501 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:27.759 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:27.801 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2022-04-10T01:27:27.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2022-04-10T01:27:27.802 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2022-04-10T01:27:27.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi043.stderr:2022-04-10T01:27:27.801+0000 7f17585a7700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 45925) UID: 1000 2022-04-10T01:27:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:27.909 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ snap_name=snap7-clean 2022-04-10T01:27:27.909 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror test snap7-clean 2022-04-10T01:27:27.909 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:27:27.909 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:27:27.910 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap7-clean 2022-04-10T01:27:27.910 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:27:27.910 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:27:27.910 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:27:28.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi043.stderr:daemon-helper: command failed with exit status 1 2022-04-10T01:27:28.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:27:28.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:29.307 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:27:29.311 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:29.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 10` 2022-04-10T01:27:29.719 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ snap_name=snap8 2022-04-10T01:27:29.719 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror test snap8 2022-04-10T01:27:29.719 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:27:29.719 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:27:29.719 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap8 2022-04-10T01:27:29.720 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:27:29.720 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:27:29.720 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:29.720 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:29.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:27:30.305 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:27:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:27:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:31.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:31.493 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:31.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ snap_name=snap8-clean 2022-04-10T01:27:31.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror test snap8-clean 2022-04-10T01:27:31.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:27:31.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:27:31.495 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap8-clean 2022-04-10T01:27:31.495 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:27:31.495 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:27:31.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:31.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:31.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:27:32.103 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:27:32.106 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:32.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:27:32.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:33.077 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:33.227 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 10` 2022-04-10T01:27:33.227 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ snap_name=snap9 2022-04-10T01:27:33.227 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror test snap9 2022-04-10T01:27:33.227 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:27:33.227 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:27:33.228 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap9 2022-04-10T01:27:33.228 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:27:33.228 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:27:33.228 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:27:33.850 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:27:33.854 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:27:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:34.905 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:35.054 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:35.054 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ snap_name=snap9-clean 2022-04-10T01:27:35.054 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror test snap9-clean 2022-04-10T01:27:35.055 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:27:35.055 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:27:35.055 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap9-clean 2022-04-10T01:27:35.056 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:27:35.056 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:27:35.056 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:35.056 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:35.057 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:27:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:27:35.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:36.392 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:27:36.396 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:36.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:36.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:36.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 10` 2022-04-10T01:27:36.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ snap_name=snap10 2022-04-10T01:27:36.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror test snap10 2022-04-10T01:27:36.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:27:36.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:27:36.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap10 2022-04-10T01:27:36.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:27:36.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:27:36.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:36.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:36.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:27:36.806 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2022-04-10T01:27:36.806 INFO:teuthology.orchestra.run:waiting for 600 2022-04-10T01:27:36.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-04-10T01:27:36.807 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2022-04-10T01:27:36.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2022-04-10T01:27:36.808 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2022-04-10T01:27:36.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2022-04-10T01:27:36.811 INFO:tasks.rbd_mirror_thrash:waiting for 113 secs before thrashing 2022-04-10T01:27:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:27:37.422 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:27:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:38.491 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:38.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ snap_name=snap10-clean 2022-04-10T01:27:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror test snap10-clean 2022-04-10T01:27:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:27:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:27:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap10-clean 2022-04-10T01:27:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:27:38.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:27:38.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:38.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:38.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10-clean /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export 2022-04-10T01:27:39.252 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:27:39.256 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10-clean /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2022-04-10T01:27:40.085 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:40.343 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-test.export /tmp/tmp.rgwz74WspX/cluster1-mirror-test.export 2022-04-10T01:27:40.491 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:40.491 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ seq 1 10 2022-04-10T01:27:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 10` 2022-04-10T01:27:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ snap_name=snap1 2022-04-10T01:27:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror test snap1 2022-04-10T01:27:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:27:40.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:27:40.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:27:40.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap1 2022-04-10T01:27:40.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:27:40.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:27:40.495 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:27:40.495 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:40.495 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1 2022-04-10T01:27:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:27:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ snap_name=snap1-clean 2022-04-10T01:27:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror test snap1-clean 2022-04-10T01:27:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:27:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:27:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:27:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap1-clean 2022-04-10T01:27:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:27:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:27:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:27:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1-clean 2022-04-10T01:27:42.291 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:27:42.310 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 10` 2022-04-10T01:27:42.311 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ snap_name=snap2 2022-04-10T01:27:42.311 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror test snap2 2022-04-10T01:27:42.311 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:27:42.311 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:27:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:27:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap2 2022-04-10T01:27:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:27:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:27:42.313 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:27:42.313 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:42.313 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2 2022-04-10T01:27:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:27:43.313 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ snap_name=snap2-clean 2022-04-10T01:27:43.313 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror test snap2-clean 2022-04-10T01:27:43.313 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:27:43.314 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:27:43.314 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:27:43.314 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap2-clean 2022-04-10T01:27:43.314 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:27:43.315 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:27:43.315 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:27:43.315 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:43.315 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2-clean 2022-04-10T01:27:44.308 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:27:44.330 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 10` 2022-04-10T01:27:44.330 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ snap_name=snap3 2022-04-10T01:27:44.331 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror test snap3 2022-04-10T01:27:44.331 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:27:44.331 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:27:44.334 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:27:44.334 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap3 2022-04-10T01:27:44.335 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:27:44.335 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:27:44.335 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:27:44.335 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:44.336 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3 2022-04-10T01:27:45.304 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:27:45.327 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ snap_name=snap3-clean 2022-04-10T01:27:45.327 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror test snap3-clean 2022-04-10T01:27:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:27:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:27:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:27:45.329 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap3-clean 2022-04-10T01:27:45.329 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:27:45.330 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:27:45.330 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:27:45.331 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:45.331 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3-clean 2022-04-10T01:27:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:27:46.344 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 10` 2022-04-10T01:27:46.344 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ snap_name=snap4 2022-04-10T01:27:46.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror test snap4 2022-04-10T01:27:46.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:27:46.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:27:46.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:27:46.346 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap4 2022-04-10T01:27:46.346 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:27:46.346 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:27:46.346 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:27:46.347 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:46.347 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4 2022-04-10T01:27:47.311 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:27:47.328 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ snap_name=snap4-clean 2022-04-10T01:27:47.328 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror test snap4-clean 2022-04-10T01:27:47.328 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:27:47.329 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:27:47.329 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:27:47.329 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap4-clean 2022-04-10T01:27:47.329 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:27:47.329 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:27:47.330 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:27:47.330 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:47.331 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4-clean 2022-04-10T01:27:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:27:48.335 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 10` 2022-04-10T01:27:48.335 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ snap_name=snap5 2022-04-10T01:27:48.335 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror test snap5 2022-04-10T01:27:48.336 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:27:48.336 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:27:48.336 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:27:48.336 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap5 2022-04-10T01:27:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:27:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:27:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:27:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5 2022-04-10T01:27:49.272 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:27:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ snap_name=snap5-clean 2022-04-10T01:27:49.293 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror test snap5-clean 2022-04-10T01:27:49.293 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:27:49.293 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:27:49.293 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:27:49.294 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap5-clean 2022-04-10T01:27:49.294 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:27:49.294 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:27:49.295 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:27:49.295 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:49.295 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5-clean 2022-04-10T01:27:50.320 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:27:50.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 10` 2022-04-10T01:27:50.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ snap_name=snap6 2022-04-10T01:27:50.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror test snap6 2022-04-10T01:27:50.346 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:27:50.346 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:27:50.346 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:27:50.346 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap6 2022-04-10T01:27:50.347 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:27:50.347 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:27:50.347 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:27:50.347 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:50.348 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6 2022-04-10T01:27:51.320 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:27:51.343 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ snap_name=snap6-clean 2022-04-10T01:27:51.343 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror test snap6-clean 2022-04-10T01:27:51.343 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:27:51.343 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:27:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:27:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap6-clean 2022-04-10T01:27:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:27:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:27:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:27:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6-clean 2022-04-10T01:27:52.322 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:27:52.340 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 10` 2022-04-10T01:27:52.340 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ snap_name=snap7 2022-04-10T01:27:52.340 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror test snap7 2022-04-10T01:27:52.341 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:27:52.341 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:27:52.341 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:27:52.341 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap7 2022-04-10T01:27:52.341 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:27:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:27:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:27:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7 2022-04-10T01:27:53.007 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2022-04-10T01:27:53.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2022-04-10T01:27:53.008 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2022-04-10T01:27:53.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr:2022-04-10T01:27:53.005+0000 7fe515e1d700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 39599) UID: 1000 2022-04-10T01:27:53.209 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr:daemon-helper: command failed with exit status 1 2022-04-10T01:27:53.322 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:27:53.337 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ snap_name=snap7-clean 2022-04-10T01:27:53.338 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror test snap7-clean 2022-04-10T01:27:53.338 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:27:53.338 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:27:53.339 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:27:53.339 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap7-clean 2022-04-10T01:27:53.339 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:27:53.339 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:27:53.339 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:27:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7-clean 2022-04-10T01:27:54.282 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:27:54.305 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 10` 2022-04-10T01:27:54.305 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ snap_name=snap8 2022-04-10T01:27:54.305 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror test snap8 2022-04-10T01:27:54.306 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:27:54.306 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:27:54.306 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:27:54.306 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap8 2022-04-10T01:27:54.307 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:27:54.307 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:27:54.307 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:27:54.307 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:54.308 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8 2022-04-10T01:27:55.333 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:27:55.350 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ snap_name=snap8-clean 2022-04-10T01:27:55.350 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror test snap8-clean 2022-04-10T01:27:55.350 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:27:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:27:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:27:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap8-clean 2022-04-10T01:27:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:27:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:27:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:27:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8-clean 2022-04-10T01:27:56.009 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2022-04-10T01:27:56.009 INFO:teuthology.orchestra.run:waiting for 600 2022-04-10T01:27:56.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-04-10T01:27:56.010 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2022-04-10T01:27:56.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2022-04-10T01:27:56.011 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2022-04-10T01:27:56.014 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2022-04-10T01:27:56.014 INFO:tasks.rbd_mirror_thrash:waiting for 69 secs before thrashing 2022-04-10T01:27:56.327 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:27:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 10` 2022-04-10T01:27:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ snap_name=snap9 2022-04-10T01:27:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror test snap9 2022-04-10T01:27:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:27:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:27:56.346 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:27:56.346 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap9 2022-04-10T01:27:56.346 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:27:56.347 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:27:56.347 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:27:56.347 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:56.347 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9 2022-04-10T01:27:57.333 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:27:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ snap_name=snap9-clean 2022-04-10T01:27:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror test snap9-clean 2022-04-10T01:27:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:27:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:27:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:27:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap9-clean 2022-04-10T01:27:57.351 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:27:57.351 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:27:57.351 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:27:57.351 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:57.352 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9-clean 2022-04-10T01:27:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:27:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 10` 2022-04-10T01:27:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ snap_name=snap10 2022-04-10T01:27:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror test snap10 2022-04-10T01:27:58.361 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:27:58.361 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:27:58.361 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:27:58.361 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap10 2022-04-10T01:27:58.362 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:27:58.362 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:27:58.362 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:27:58.362 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10 2022-04-10T01:27:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:27:59.282 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ snap_name=snap10-clean 2022-04-10T01:27:59.283 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror test snap10-clean 2022-04-10T01:27:59.283 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:27:59.283 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:27:59.283 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:27:59.284 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap10-clean 2022-04-10T01:27:59.284 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:27:59.284 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:27:59.284 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:27:59.285 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:27:59.285 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10-clean 2022-04-10T01:28:00.341 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:28:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror test 2022-04-10T01:28:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:28:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:28:00.352 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:28:00.352 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:28:00.352 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:28:00.352 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:28:00.353 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:28:00.353 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:28:00.353 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:28:00.354 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror test 2022-04-10T01:28:00.354 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:28:00.355 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:28:00.355 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:28:00.355 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:28:00.355 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:28:00.355 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:28:00.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:28:00.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2022-04-10T01:28:00.402 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/test 2022-04-10T01:28:00.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi043.stderr:2022-04-10T01:28:00.514+0000 7fa7b8014700 -1 rbd::mirror::ImageMap: 0x562f233bcc60 handle_peer_ack_remove: failed to remove global_image_id=5ff848f4-a6a0-498d-b9f1-7a041de7de98 2022-04-10T01:28:01.648 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2022-04-10T01:28:01.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:28:01.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror test deleted 2022-04-10T01:28:01.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:28:01.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:28:01.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:28:01.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:28:01.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:28:01.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:28:01.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:28:01.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:28:01.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:28:01.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:28:01.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:28:01.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror test 2022-04-10T01:28:01.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:28:01.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:28:01.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=test 2022-04-10T01:28:01.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:28:01.658 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:28:01.658 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:28:01.658 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:28:01.658 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/test 2022-04-10T01:28:01.659 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:28:01.681 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image test: (2) No such file or directory 2022-04-10T01:28:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:28:01.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:28:01.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:28:01.787 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror test deleted '' 2022-04-10T01:28:01.787 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:28:01.787 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:28:01.788 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=test 2022-04-10T01:28:01.788 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:28:01.788 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:28:01.788 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:28:01.789 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:28:01.789 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:28:01.789 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:28:01.789 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:28:01.790 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:28:01.790 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror test 2022-04-10T01:28:01.790 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:28:01.790 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:28:01.791 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=test 2022-04-10T01:28:01.791 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:28:01.791 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:28:01.791 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:28:01.792 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:28:01.792 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/test 2022-04-10T01:28:01.792 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:28:01.819 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image test: (2) No such file or directory 2022-04-10T01:28:01.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:28:01.823 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:28:01.823 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:28:01.823 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:28:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ testlog 'TEST: create many images' 2022-04-10T01:28:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ tee -a /tmp/tmp.rgwz74WspX/rbd-mirror.test.log 2022-04-10T01:28:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ date '+%F %T' 2022-04-10T01:28:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo 2022-04-10 01:28:01 TEST: create many images 2022-04-10T01:28:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:2022-04-10 01:28:01 TEST: create many images 2022-04-10T01:28:01.828 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ snap_name=snap 2022-04-10T01:28:01.828 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ seq 1 50 2022-04-10T01:28:01.828 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:28:01.828 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_1 2022-04-10T01:28:01.829 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image_and_enable_mirror cluster2 mirror image_1 snapshot 128M 2022-04-10T01:28:01.829 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:28:01.829 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:28:01.829 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:28:01.830 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:28:01.830 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_1 2022-04-10T01:28:01.830 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:28:01.830 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:28:01.831 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:28:01.831 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:28:01.831 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:28:01.831 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:28:01.831 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snapshot ']' 2022-04-10T01:28:01.832 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:28:01.832 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_1 128M 2022-04-10T01:28:01.832 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:28:01.832 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:28:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:28:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:28:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_1 2022-04-10T01:28:01.834 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:28:01.834 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:28:01.834 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:28:01.834 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:28:01.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:28:01.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:28:01.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 128M ']' 2022-04-10T01:28:01.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=128M 2022-04-10T01:28:01.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:28:01.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_1 2022-04-10T01:28:01.872 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' image = image ']' 2022-04-10T01:28:01.872 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_1 snapshot 2022-04-10T01:28:01.872 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:28:01.873 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:28:01.873 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_1 2022-04-10T01:28:01.873 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:28:01.873 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:28:01.874 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:28:01.874 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:28:01.874 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:28:01.874 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2022-04-10T01:28:02.351 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:28:02.361 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2022-04-10T01:28:02.393 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_1': 2022-04-10T01:28:02.394 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:28:02.394 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:28:02.394 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:28:02.395 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 1255b186d842 2022-04-10T01:28:02.395 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.1255b186d842 2022-04-10T01:28:02.395 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:28:02.395 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:28:02.396 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:28:02.396 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:28:02.396 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:28:01 2022 2022-04-10T01:28:02.396 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:28:01 2022 2022-04-10T01:28:02.397 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:28:01 2022 2022-04-10T01:28:02.397 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:28:02.397 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:28:02.397 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: acfa4173-7a18-4ae9-975a-94af9fdf80e4 2022-04-10T01:28:02.398 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:28:02.399 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:28:02.400 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_1 2022-04-10T01:28:02.400 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:28:02.400 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:28:02.400 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_1 2022-04-10T01:28:02.400 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:28:02.401 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=6 2022-04-10T01:28:02.401 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:28:02.402 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:28:02.402 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:28:02.402 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:28:02.402 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:28:02.403 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:28:02.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 27985920 bytes 2022-04-10T01:28:02.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 71250944 bytes 2022-04-10T01:28:02.930 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 110570496 bytes 2022-04-10T01:28:03.101 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 144527360 bytes 2022-04-10T01:28:03.316 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 217329664 bytes 2022-04-10T01:28:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 324273152 bytes 2022-04-10T01:28:03.798 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 401133568 bytes 2022-04-10T01:28:03.988 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 451597312 bytes 2022-04-10T01:28:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 471568384 bytes 2022-04-10T01:28:04.348 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 512037888 bytes 2022-04-10T01:28:04.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 532460544 bytes 2022-04-10T01:28:04.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 572592128 bytes 2022-04-10T01:28:04.870 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 626422784 bytes 2022-04-10T01:28:05.045 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 666175488 bytes 2022-04-10T01:28:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 707129344 bytes 2022-04-10T01:28:05.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 729990144 bytes 2022-04-10T01:28:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 761444352 bytes 2022-04-10T01:28:05.698 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 778838016 bytes 2022-04-10T01:28:05.879 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 820960256 bytes 2022-04-10T01:28:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 868611072 bytes 2022-04-10T01:28:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 938068992 bytes 2022-04-10T01:28:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 955737088 bytes 2022-04-10T01:28:06.587 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 979119104 bytes 2022-04-10T01:28:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 1000754176 bytes 2022-04-10T01:28:06.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 1067455488 bytes 2022-04-10T01:28:07.093 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1079663616 bytes 2022-04-10T01:28:07.277 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1118462976 bytes 2022-04-10T01:28:07.446 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1148759040 bytes 2022-04-10T01:28:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1185792000 bytes 2022-04-10T01:28:07.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1227657216 bytes 2022-04-10T01:28:07.989 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1269776384 bytes 2022-04-10T01:28:08.167 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1299482624 bytes 2022-04-10T01:28:08.404 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:28:08.405 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:28:08.405 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:28:08.406 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:28:08.406 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:28:08.407 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_2 2022-04-10T01:28:08.407 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image_and_enable_mirror cluster2 mirror image_2 snapshot 128M 2022-04-10T01:28:08.408 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:28:08.408 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:28:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:28:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:28:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_2 2022-04-10T01:28:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:28:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:28:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:28:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:28:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:28:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:28:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snapshot ']' 2022-04-10T01:28:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:28:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_2 128M 2022-04-10T01:28:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:28:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:28:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:28:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:28:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_2 2022-04-10T01:28:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:28:08.417 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:28:08.417 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:28:08.418 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:28:08.418 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:28:08.419 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:28:08.419 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 128M ']' 2022-04-10T01:28:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=128M 2022-04-10T01:28:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:28:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_2 2022-04-10T01:28:08.452 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' image = image ']' 2022-04-10T01:28:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_2 snapshot 2022-04-10T01:28:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:28:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:28:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_2 2022-04-10T01:28:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:28:08.455 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:28:08.455 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:28:08.456 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:28:08.456 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:28:08.457 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2022-04-10T01:28:09.304 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:28:09.315 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2022-04-10T01:28:09.349 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_2': 2022-04-10T01:28:09.349 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:28:09.349 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:28:09.349 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:28:09.350 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 125dcf2ce2c 2022-04-10T01:28:09.350 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.125dcf2ce2c 2022-04-10T01:28:09.350 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:28:09.350 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:28:09.351 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:28:09.351 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:28:09.351 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:28:08 2022 2022-04-10T01:28:09.351 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:28:08 2022 2022-04-10T01:28:09.351 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:28:08 2022 2022-04-10T01:28:09.352 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:28:09.352 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:28:09.352 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: a0c16b90-9fb2-4d3c-834f-4b590db7c01d 2022-04-10T01:28:09.352 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:28:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:28:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_2 2022-04-10T01:28:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:28:09.355 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:28:09.355 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_2 2022-04-10T01:28:09.355 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:28:09.355 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=10 2022-04-10T01:28:09.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:28:09.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:28:09.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:28:09.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:28:09.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:28:09.357 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:28:09.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 20666368 bytes 2022-04-10T01:28:09.726 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 64765952 bytes 2022-04-10T01:28:09.883 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 89907200 bytes 2022-04-10T01:28:10.048 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 126016512 bytes 2022-04-10T01:28:10.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 179777536 bytes 2022-04-10T01:28:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 245025792 bytes 2022-04-10T01:28:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 260253696 bytes 2022-04-10T01:28:10.747 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 285230080 bytes 2022-04-10T01:28:10.962 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 359096320 bytes 2022-04-10T01:28:11.145 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 409187328 bytes 2022-04-10T01:28:11.376 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 501634048 bytes 2022-04-10T01:28:11.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 548726784 bytes 2022-04-10T01:28:11.780 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 616501248 bytes 2022-04-10T01:28:11.978 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 675658752 bytes 2022-04-10T01:28:12.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 745829376 bytes 2022-04-10T01:28:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 832143360 bytes 2022-04-10T01:28:12.616 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 867567616 bytes 2022-04-10T01:28:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 922689536 bytes 2022-04-10T01:28:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 991980544 bytes 2022-04-10T01:28:13.286 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 1090282496 bytes 2022-04-10T01:28:13.462 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 1123596288 bytes 2022-04-10T01:28:13.650 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 1176293376 bytes 2022-04-10T01:28:13.832 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 1223656448 bytes 2022-04-10T01:28:14.032 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 1278135296 bytes 2022-04-10T01:28:14.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 1309261824 bytes 2022-04-10T01:28:14.433 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1383707648 bytes 2022-04-10T01:28:14.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1401042944 bytes 2022-04-10T01:28:14.724 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1412939776 bytes 2022-04-10T01:28:15.007 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1489914880 bytes 2022-04-10T01:28:15.234 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1538039808 bytes 2022-04-10T01:28:15.445 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1595868160 bytes 2022-04-10T01:28:15.601 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1617240064 bytes 2022-04-10T01:28:15.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1641777152 bytes 2022-04-10T01:28:15.968 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1697472512 bytes 2022-04-10T01:28:16.177 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1767611392 bytes 2022-04-10T01:28:16.445 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1884157952 bytes 2022-04-10T01:28:16.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1956530176 bytes 2022-04-10T01:28:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 2042524672 bytes 2022-04-10T01:28:17.077 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 2069801984 bytes 2022-04-10T01:28:17.227 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 2086227968 bytes 2022-04-10T01:28:17.407 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 2119839744 bytes 2022-04-10T01:28:17.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 2132457472 bytes 2022-04-10T01:28:17.739 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 2193696768 bytes 2022-04-10T01:28:17.904 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 2205846528 bytes 2022-04-10T01:28:18.147 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 2302050304 bytes 2022-04-10T01:28:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 2318796800 bytes 2022-04-10T01:28:18.606 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 2436605952 bytes 2022-04-10T01:28:18.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 2486996992 bytes 2022-04-10T01:28:18.952 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 2517346304 bytes 2022-04-10T01:28:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 2534771712 bytes 2022-04-10T01:28:19.263 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 2536248320 bytes 2022-04-10T01:28:19.365 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:28:19.365 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:28:19.365 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:28:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:28:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:28:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_3 2022-04-10T01:28:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image_and_enable_mirror cluster2 mirror image_3 snapshot 128M 2022-04-10T01:28:19.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:28:19.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:28:19.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:28:19.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:28:19.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_3 2022-04-10T01:28:19.368 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:28:19.368 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:28:19.368 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:28:19.368 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:28:19.369 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:28:19.369 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:28:19.369 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snapshot ']' 2022-04-10T01:28:19.369 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:28:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_3 128M 2022-04-10T01:28:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:28:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:28:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:28:19.371 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:28:19.371 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_3 2022-04-10T01:28:19.371 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:28:19.371 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:28:19.371 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:28:19.372 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:28:19.372 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:28:19.372 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:28:19.372 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 128M ']' 2022-04-10T01:28:19.372 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=128M 2022-04-10T01:28:19.373 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:28:19.373 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_3 2022-04-10T01:28:19.409 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' image = image ']' 2022-04-10T01:28:19.410 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_3 snapshot 2022-04-10T01:28:19.410 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:28:19.410 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:28:19.411 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_3 2022-04-10T01:28:19.411 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:28:19.411 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:28:19.411 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:28:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:28:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:28:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2022-04-10T01:28:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:28:20.312 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2022-04-10T01:28:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_3': 2022-04-10T01:28:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:28:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:28:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:28:20.349 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 1266a8d2568b 2022-04-10T01:28:20.349 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.1266a8d2568b 2022-04-10T01:28:20.349 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:28:20.350 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:28:20.350 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:28:20.350 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:28:20.351 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:28:19 2022 2022-04-10T01:28:20.351 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:28:19 2022 2022-04-10T01:28:20.351 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:28:19 2022 2022-04-10T01:28:20.351 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:28:20.351 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:28:20.352 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 6d595122-4619-4d21-a653-5b0cde552987 2022-04-10T01:28:20.352 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:28:20.355 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:28:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_3 2022-04-10T01:28:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:28:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:28:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_3 2022-04-10T01:28:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:28:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=13 2022-04-10T01:28:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:28:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:28:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:28:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:28:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:28:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:28:20.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 24792064 bytes 2022-04-10T01:28:20.750 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 82283520 bytes 2022-04-10T01:28:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 95713280 bytes 2022-04-10T01:28:21.049 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 114440192 bytes 2022-04-10T01:28:21.279 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 195186688 bytes 2022-04-10T01:28:21.454 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 222801920 bytes 2022-04-10T01:28:21.650 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 276803584 bytes 2022-04-10T01:28:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 294617088 bytes 2022-04-10T01:28:22.070 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 392559616 bytes 2022-04-10T01:28:22.260 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 455513088 bytes 2022-04-10T01:28:22.465 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 493112320 bytes 2022-04-10T01:28:22.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 522583040 bytes 2022-04-10T01:28:22.847 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 587151360 bytes 2022-04-10T01:28:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 623560704 bytes 2022-04-10T01:28:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 652481536 bytes 2022-04-10T01:28:23.348 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 659378176 bytes 2022-04-10T01:28:23.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 719639552 bytes 2022-04-10T01:28:23.741 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 760977408 bytes 2022-04-10T01:28:23.969 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 840370176 bytes 2022-04-10T01:28:24.105 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 840982528 bytes 2022-04-10T01:28:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 887778304 bytes 2022-04-10T01:28:24.446 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 888384512 bytes 2022-04-10T01:28:24.660 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 949812224 bytes 2022-04-10T01:28:24.842 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 987267072 bytes 2022-04-10T01:28:25.032 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 1032114176 bytes 2022-04-10T01:28:25.197 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1055089664 bytes 2022-04-10T01:28:25.450 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1166489600 bytes 2022-04-10T01:28:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1209244672 bytes 2022-04-10T01:28:25.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1209873408 bytes 2022-04-10T01:28:25.945 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1214546944 bytes 2022-04-10T01:28:26.139 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1262935040 bytes 2022-04-10T01:28:26.287 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1273604096 bytes 2022-04-10T01:28:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1314679808 bytes 2022-04-10T01:28:26.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1360055296 bytes 2022-04-10T01:28:26.862 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1419058176 bytes 2022-04-10T01:28:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1464397824 bytes 2022-04-10T01:28:27.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1493439488 bytes 2022-04-10T01:28:27.405 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1513445376 bytes 2022-04-10T01:28:27.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1602190336 bytes 2022-04-10T01:28:27.903 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1705289728 bytes 2022-04-10T01:28:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1776491520 bytes 2022-04-10T01:28:28.382 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 1845710848 bytes 2022-04-10T01:28:28.557 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 1876819968 bytes 2022-04-10T01:28:28.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 1877438464 bytes 2022-04-10T01:28:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 1925426176 bytes 2022-04-10T01:28:29.090 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 1972534272 bytes 2022-04-10T01:28:29.265 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 2007976960 bytes 2022-04-10T01:28:29.439 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 2037513216 bytes 2022-04-10T01:28:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 2059344896 bytes 2022-04-10T01:28:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 2100453376 bytes 2022-04-10T01:28:30.007 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 2167341056 bytes 2022-04-10T01:28:30.172 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 2192612352 bytes 2022-04-10T01:28:30.352 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 2227674112 bytes 2022-04-10T01:28:30.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5400 IOs, 2258893824 bytes 2022-04-10T01:28:30.706 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5500 IOs, 2309555200 bytes 2022-04-10T01:28:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5600 IOs, 2397688832 bytes 2022-04-10T01:28:31.169 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5700 IOs, 2451966976 bytes 2022-04-10T01:28:31.322 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5800 IOs, 2463369216 bytes 2022-04-10T01:28:31.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5900 IOs, 2530817024 bytes 2022-04-10T01:28:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6000 IOs, 2606120960 bytes 2022-04-10T01:28:31.980 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6100 IOs, 2652566528 bytes 2022-04-10T01:28:32.243 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6200 IOs, 2750964736 bytes 2022-04-10T01:28:32.406 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6300 IOs, 2764226560 bytes 2022-04-10T01:28:32.583 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6400 IOs, 2788808704 bytes 2022-04-10T01:28:32.731 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6500 IOs, 2800017408 bytes 2022-04-10T01:28:32.926 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6600 IOs, 2837954560 bytes 2022-04-10T01:28:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6700 IOs, 2863266816 bytes 2022-04-10T01:28:33.277 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6800 IOs, 2896937984 bytes 2022-04-10T01:28:33.364 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:28:33.365 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:28:33.365 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:28:33.366 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:28:33.366 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:28:33.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_4 2022-04-10T01:28:33.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image_and_enable_mirror cluster2 mirror image_4 snapshot 128M 2022-04-10T01:28:33.368 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:28:33.368 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:28:33.368 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:28:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:28:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_4 2022-04-10T01:28:33.370 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:28:33.370 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:28:33.371 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:28:33.371 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:28:33.371 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:28:33.372 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:28:33.372 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snapshot ']' 2022-04-10T01:28:33.373 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:28:33.373 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_4 128M 2022-04-10T01:28:33.374 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:28:33.374 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:28:33.375 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:28:33.375 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:28:33.375 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_4 2022-04-10T01:28:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:28:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:28:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:28:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:28:33.378 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:28:33.378 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:28:33.378 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 128M ']' 2022-04-10T01:28:33.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=128M 2022-04-10T01:28:33.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:28:33.380 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_4 2022-04-10T01:28:33.406 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' image = image ']' 2022-04-10T01:28:33.407 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_4 snapshot 2022-04-10T01:28:33.407 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:28:33.407 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:28:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_4 2022-04-10T01:28:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:28:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:28:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:28:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:28:33.409 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:28:33.409 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2022-04-10T01:28:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:28:34.311 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2022-04-10T01:28:34.347 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_4': 2022-04-10T01:28:34.347 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:28:34.347 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:28:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:28:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 126f4a560cc3 2022-04-10T01:28:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.126f4a560cc3 2022-04-10T01:28:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:28:34.349 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:28:34.349 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:28:34.349 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:28:34.349 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:28:33 2022 2022-04-10T01:28:34.349 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:28:33 2022 2022-04-10T01:28:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:28:33 2022 2022-04-10T01:28:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:28:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:28:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: d4a4b534-85cd-4706-8f4e-137718b81a97 2022-04-10T01:28:34.351 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:28:34.352 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:28:34.352 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_4 2022-04-10T01:28:34.353 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:28:34.353 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:28:34.353 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_4 2022-04-10T01:28:34.353 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:28:34.353 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=13 2022-04-10T01:28:34.354 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:28:34.354 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:28:34.354 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:28:34.354 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:28:34.355 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:28:34.355 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:28:34.544 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 10317824 bytes 2022-04-10T01:28:34.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 45029376 bytes 2022-04-10T01:28:34.876 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 73393152 bytes 2022-04-10T01:28:35.059 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 113944576 bytes 2022-04-10T01:28:35.228 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 143876096 bytes 2022-04-10T01:28:35.373 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 153447424 bytes 2022-04-10T01:28:35.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 250570752 bytes 2022-04-10T01:28:35.796 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 267140096 bytes 2022-04-10T01:28:36.022 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 347635712 bytes 2022-04-10T01:28:36.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 391920640 bytes 2022-04-10T01:28:36.383 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 423617536 bytes 2022-04-10T01:28:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 460075008 bytes 2022-04-10T01:28:36.745 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 483733504 bytes 2022-04-10T01:28:36.903 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 516402176 bytes 2022-04-10T01:28:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 574652416 bytes 2022-04-10T01:28:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 608787456 bytes 2022-04-10T01:28:37.465 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 639210496 bytes 2022-04-10T01:28:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 715913216 bytes 2022-04-10T01:28:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 759307264 bytes 2022-04-10T01:28:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 799005696 bytes 2022-04-10T01:28:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 807713792 bytes 2022-04-10T01:28:38.408 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 830597120 bytes 2022-04-10T01:28:38.609 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 879984640 bytes 2022-04-10T01:28:38.784 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 908968960 bytes 2022-04-10T01:28:38.972 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 951703552 bytes 2022-04-10T01:28:39.174 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 996839424 bytes 2022-04-10T01:28:39.319 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 997431296 bytes 2022-04-10T01:28:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1065595904 bytes 2022-04-10T01:28:39.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1092670464 bytes 2022-04-10T01:28:39.913 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1132148736 bytes 2022-04-10T01:28:40.112 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1186096128 bytes 2022-04-10T01:28:40.269 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1224653824 bytes 2022-04-10T01:28:40.486 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1265146880 bytes 2022-04-10T01:28:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1269777408 bytes 2022-04-10T01:28:40.845 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1322992640 bytes 2022-04-10T01:28:41.026 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1354431488 bytes 2022-04-10T01:28:41.260 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1452067840 bytes 2022-04-10T01:28:41.441 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1470337024 bytes 2022-04-10T01:28:41.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1566192640 bytes 2022-04-10T01:28:41.839 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1566801920 bytes 2022-04-10T01:28:42.073 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1636823040 bytes 2022-04-10T01:28:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 1637425152 bytes 2022-04-10T01:28:42.376 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 1674415104 bytes 2022-04-10T01:28:42.605 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 1717768192 bytes 2022-04-10T01:28:42.790 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 1751334912 bytes 2022-04-10T01:28:43.014 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 1805819904 bytes 2022-04-10T01:28:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 1891382272 bytes 2022-04-10T01:28:43.415 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 1900611584 bytes 2022-04-10T01:28:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 1941536768 bytes 2022-04-10T01:28:43.778 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 1962518528 bytes 2022-04-10T01:28:43.982 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 1980706816 bytes 2022-04-10T01:28:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 2049558528 bytes 2022-04-10T01:28:44.436 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 2094029824 bytes 2022-04-10T01:28:44.702 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5400 IOs, 2191183872 bytes 2022-04-10T01:28:44.891 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5500 IOs, 2234801152 bytes 2022-04-10T01:28:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5600 IOs, 2277278720 bytes 2022-04-10T01:28:45.258 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5700 IOs, 2300021760 bytes 2022-04-10T01:28:45.433 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5800 IOs, 2325831680 bytes 2022-04-10T01:28:45.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5900 IOs, 2380326912 bytes 2022-04-10T01:28:45.881 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6000 IOs, 2447303680 bytes 2022-04-10T01:28:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6100 IOs, 2484969472 bytes 2022-04-10T01:28:46.276 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6200 IOs, 2544444416 bytes 2022-04-10T01:28:46.491 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6300 IOs, 2606265344 bytes 2022-04-10T01:28:46.724 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6400 IOs, 2676135936 bytes 2022-04-10T01:28:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6500 IOs, 2726582272 bytes 2022-04-10T01:28:47.142 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6600 IOs, 2778982400 bytes 2022-04-10T01:28:47.362 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:28:47.363 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:28:47.363 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:28:47.364 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:28:47.364 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:28:47.364 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_5 2022-04-10T01:28:47.364 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image_and_enable_mirror cluster2 mirror image_5 snapshot 128M 2022-04-10T01:28:47.364 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:28:47.365 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:28:47.365 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:28:47.365 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:28:47.365 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_5 2022-04-10T01:28:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:28:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:28:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:28:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:28:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:28:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:28:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snapshot ']' 2022-04-10T01:28:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:28:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_5 128M 2022-04-10T01:28:47.368 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:28:47.368 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:28:47.368 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:28:47.368 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:28:47.369 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_5 2022-04-10T01:28:47.369 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:28:47.369 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:28:47.369 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:28:47.369 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:28:47.370 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:28:47.370 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:28:47.370 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 128M ']' 2022-04-10T01:28:47.370 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=128M 2022-04-10T01:28:47.371 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:28:47.371 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_5 2022-04-10T01:28:47.410 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' image = image ']' 2022-04-10T01:28:47.410 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_5 snapshot 2022-04-10T01:28:47.411 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:28:47.411 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:28:47.411 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_5 2022-04-10T01:28:47.411 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:28:47.412 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:28:47.412 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:28:47.412 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:28:47.412 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:28:47.412 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2022-04-10T01:28:48.316 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:28:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2022-04-10T01:28:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_5': 2022-04-10T01:28:48.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:28:48.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:28:48.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:28:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 127778847211 2022-04-10T01:28:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.127778847211 2022-04-10T01:28:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:28:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:28:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:28:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:28:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:28:47 2022 2022-04-10T01:28:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:28:47 2022 2022-04-10T01:28:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:28:47 2022 2022-04-10T01:28:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:28:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:28:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 04a20513-9ad9-463b-ad2e-0f637bb8e1de 2022-04-10T01:28:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:28:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:28:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_5 2022-04-10T01:28:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:28:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:28:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_5 2022-04-10T01:28:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:28:48.374 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=11 2022-04-10T01:28:48.374 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:28:48.374 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:28:48.374 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:28:48.375 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:28:48.375 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:28:48.375 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:28:48.591 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 29765632 bytes 2022-04-10T01:28:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 89815040 bytes 2022-04-10T01:28:48.956 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 102005760 bytes 2022-04-10T01:28:49.134 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 132414464 bytes 2022-04-10T01:28:49.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 199715840 bytes 2022-04-10T01:28:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 243658752 bytes 2022-04-10T01:28:49.738 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 272251904 bytes 2022-04-10T01:28:49.927 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 309335040 bytes 2022-04-10T01:28:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 333488128 bytes 2022-04-10T01:28:50.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 381748224 bytes 2022-04-10T01:28:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 421807104 bytes 2022-04-10T01:28:50.931 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 516603904 bytes 2022-04-10T01:28:51.155 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 586570752 bytes 2022-04-10T01:28:51.322 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 610252800 bytes 2022-04-10T01:28:51.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 646493184 bytes 2022-04-10T01:28:51.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 666376192 bytes 2022-04-10T01:28:51.863 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 697157632 bytes 2022-04-10T01:28:52.009 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 697732096 bytes 2022-04-10T01:28:52.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 738926592 bytes 2022-04-10T01:28:52.430 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 810886144 bytes 2022-04-10T01:28:52.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 811500544 bytes 2022-04-10T01:28:52.736 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 834065408 bytes 2022-04-10T01:28:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 870112256 bytes 2022-04-10T01:28:53.116 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 933568512 bytes 2022-04-10T01:28:53.341 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 979418112 bytes 2022-04-10T01:28:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1010589696 bytes 2022-04-10T01:28:53.702 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1046569984 bytes 2022-04-10T01:28:53.923 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1087005696 bytes 2022-04-10T01:28:54.095 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1125618688 bytes 2022-04-10T01:28:54.296 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1163481088 bytes 2022-04-10T01:28:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1177907200 bytes 2022-04-10T01:28:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1272505344 bytes 2022-04-10T01:28:54.884 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1307164672 bytes 2022-04-10T01:28:55.122 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1371579392 bytes 2022-04-10T01:28:55.323 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1421411328 bytes 2022-04-10T01:28:55.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1434105856 bytes 2022-04-10T01:28:55.745 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1527324672 bytes 2022-04-10T01:28:56.018 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1646398464 bytes 2022-04-10T01:28:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1714390016 bytes 2022-04-10T01:28:56.455 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1760152576 bytes 2022-04-10T01:28:56.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1825193984 bytes 2022-04-10T01:28:56.886 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 1859772416 bytes 2022-04-10T01:28:57.064 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 1889043456 bytes 2022-04-10T01:28:57.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 1910198272 bytes 2022-04-10T01:28:57.400 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 1920243712 bytes 2022-04-10T01:28:57.614 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 1980165120 bytes 2022-04-10T01:28:57.881 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 2074060800 bytes 2022-04-10T01:28:58.034 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 2095193088 bytes 2022-04-10T01:28:58.247 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 2145555456 bytes 2022-04-10T01:28:58.437 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 2184899584 bytes 2022-04-10T01:28:58.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 2238386176 bytes 2022-04-10T01:28:58.809 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 2258122752 bytes 2022-04-10T01:28:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 2258740224 bytes 2022-04-10T01:28:59.158 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5400 IOs, 2306928640 bytes 2022-04-10T01:28:59.346 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5500 IOs, 2350091264 bytes 2022-04-10T01:28:59.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:28:59.380 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:28:59.380 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:28:59.380 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:28:59.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:28:59.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_6 2022-04-10T01:28:59.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image_and_enable_mirror cluster2 mirror image_6 snapshot 128M 2022-04-10T01:28:59.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:28:59.382 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:28:59.382 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:28:59.382 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:28:59.382 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_6 2022-04-10T01:28:59.383 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:28:59.383 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:28:59.383 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:28:59.383 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:28:59.384 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:28:59.384 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:28:59.384 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snapshot ']' 2022-04-10T01:28:59.384 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:28:59.385 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_6 128M 2022-04-10T01:28:59.385 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:28:59.385 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:28:59.385 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:28:59.386 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:28:59.386 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_6 2022-04-10T01:28:59.386 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:28:59.386 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:28:59.386 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:28:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:28:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:28:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:28:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 128M ']' 2022-04-10T01:28:59.388 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=128M 2022-04-10T01:28:59.388 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:28:59.388 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_6 2022-04-10T01:28:59.426 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' image = image ']' 2022-04-10T01:28:59.426 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_6 snapshot 2022-04-10T01:28:59.427 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:28:59.427 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:28:59.427 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_6 2022-04-10T01:28:59.427 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:28:59.427 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:28:59.428 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:28:59.428 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:28:59.428 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:28:59.428 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2022-04-10T01:29:00.315 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:29:00.327 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2022-04-10T01:29:00.363 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_6': 2022-04-10T01:29:00.363 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:29:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:29:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:29:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 12806d94b674 2022-04-10T01:29:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.12806d94b674 2022-04-10T01:29:00.365 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:29:00.365 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:29:00.365 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:29:00.365 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:29:00.365 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:28:59 2022 2022-04-10T01:29:00.366 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:28:59 2022 2022-04-10T01:29:00.366 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:28:59 2022 2022-04-10T01:29:00.366 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:29:00.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:29:00.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: a2f508f1-f607-46da-a720-0b026055c5d9 2022-04-10T01:29:00.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:29:00.369 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:29:00.369 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_6 2022-04-10T01:29:00.369 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:29:00.370 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:29:00.370 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_6 2022-04-10T01:29:00.370 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:29:00.370 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=13 2022-04-10T01:29:00.370 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:29:00.371 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:29:00.371 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:29:00.371 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:29:00.371 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:29:00.372 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:29:00.586 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 40983552 bytes 2022-04-10T01:29:00.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 72939520 bytes 2022-04-10T01:29:00.979 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 131201024 bytes 2022-04-10T01:29:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 161221632 bytes 2022-04-10T01:29:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 234230784 bytes 2022-04-10T01:29:01.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 270357504 bytes 2022-04-10T01:29:01.715 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 278813696 bytes 2022-04-10T01:29:01.915 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 321636352 bytes 2022-04-10T01:29:02.166 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 417337344 bytes 2022-04-10T01:29:02.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 594992128 bytes 2022-04-10T01:29:02.701 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 619912192 bytes 2022-04-10T01:29:02.956 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 714985472 bytes 2022-04-10T01:29:03.171 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 782959616 bytes 2022-04-10T01:29:03.322 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 788670464 bytes 2022-04-10T01:29:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 811214848 bytes 2022-04-10T01:29:03.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 863640576 bytes 2022-04-10T01:29:03.851 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 880990208 bytes 2022-04-10T01:29:04.072 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 944358400 bytes 2022-04-10T01:29:04.270 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 998038528 bytes 2022-04-10T01:29:04.463 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 1040443392 bytes 2022-04-10T01:29:04.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 1083061248 bytes 2022-04-10T01:29:04.846 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 1132013568 bytes 2022-04-10T01:29:05.015 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.349194489831938) > weight (0.25) 2022-04-10T01:29:05.016 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2022-04-10T01:29:05.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2022-04-10T01:29:05.017 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2022-04-10T01:29:05.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi043.stderr:2022-04-10T01:29:05.015+0000 7fa7b9016700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 39643) UID: 1000 2022-04-10T01:29:05.071 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 1176274944 bytes 2022-04-10T01:29:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 1207599104 bytes 2022-04-10T01:29:05.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi043.stderr:daemon-helper: command failed with exit status 1 2022-04-10T01:29:05.491 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 1271704576 bytes 2022-04-10T01:29:05.659 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1290077184 bytes 2022-04-10T01:29:05.833 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1314206720 bytes 2022-04-10T01:29:06.017 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1340410880 bytes 2022-04-10T01:29:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1359644672 bytes 2022-04-10T01:29:06.388 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1405685760 bytes 2022-04-10T01:29:06.556 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1430040576 bytes 2022-04-10T01:29:06.727 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1455528960 bytes 2022-04-10T01:29:06.903 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1480634368 bytes 2022-04-10T01:29:07.133 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1542242304 bytes 2022-04-10T01:29:07.327 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1596090368 bytes 2022-04-10T01:29:07.492 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1610011648 bytes 2022-04-10T01:29:07.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1633186816 bytes 2022-04-10T01:29:07.882 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1694708736 bytes 2022-04-10T01:29:08.104 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1760355328 bytes 2022-04-10T01:29:08.305 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1793024000 bytes 2022-04-10T01:29:08.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1841895424 bytes 2022-04-10T01:29:08.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 1863513088 bytes 2022-04-10T01:29:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 1927287808 bytes 2022-04-10T01:29:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 1935242240 bytes 2022-04-10T01:29:09.244 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 1974492160 bytes 2022-04-10T01:29:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 2040448000 bytes 2022-04-10T01:29:09.606 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 2043366400 bytes 2022-04-10T01:29:09.856 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 2132900864 bytes 2022-04-10T01:29:10.012 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 2135855104 bytes 2022-04-10T01:29:10.018 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2022-04-10T01:29:10.019 INFO:teuthology.orchestra.run:waiting for 600 2022-04-10T01:29:10.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-04-10T01:29:10.020 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2022-04-10T01:29:10.020 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2022-04-10T01:29:10.020 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2022-04-10T01:29:10.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2022-04-10T01:29:10.023 INFO:tasks.rbd_mirror_thrash:waiting for 111 secs before thrashing 2022-04-10T01:29:10.180 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 2140797952 bytes 2022-04-10T01:29:10.377 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 2161983488 bytes 2022-04-10T01:29:10.625 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 2235972608 bytes 2022-04-10T01:29:10.786 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 2242946048 bytes 2022-04-10T01:29:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5400 IOs, 2279933952 bytes 2022-04-10T01:29:11.167 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5500 IOs, 2309555200 bytes 2022-04-10T01:29:11.387 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5600 IOs, 2372430848 bytes 2022-04-10T01:29:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5700 IOs, 2373047296 bytes 2022-04-10T01:29:11.739 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5800 IOs, 2395799552 bytes 2022-04-10T01:29:11.926 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5900 IOs, 2439294976 bytes 2022-04-10T01:29:12.179 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6000 IOs, 2505073664 bytes 2022-04-10T01:29:12.330 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6100 IOs, 2507719680 bytes 2022-04-10T01:29:12.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6200 IOs, 2544906240 bytes 2022-04-10T01:29:12.744 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6300 IOs, 2595794944 bytes 2022-04-10T01:29:12.931 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6400 IOs, 2624676864 bytes 2022-04-10T01:29:13.160 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6500 IOs, 2650732544 bytes 2022-04-10T01:29:13.363 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6600 IOs, 2685598720 bytes 2022-04-10T01:29:13.376 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:29:13.376 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:29:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:29:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:29:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:29:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_7 2022-04-10T01:29:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image_and_enable_mirror cluster2 mirror image_7 snapshot 128M 2022-04-10T01:29:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:29:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:29:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:29:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:29:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_7 2022-04-10T01:29:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:29:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:29:13.380 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:29:13.380 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:29:13.380 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:29:13.380 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:29:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snapshot ']' 2022-04-10T01:29:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:29:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_7 128M 2022-04-10T01:29:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:29:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:29:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:29:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:29:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_7 2022-04-10T01:29:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:29:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:29:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:29:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:29:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:29:13.384 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:29:13.384 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 128M ']' 2022-04-10T01:29:13.384 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=128M 2022-04-10T01:29:13.384 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:29:13.385 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_7 2022-04-10T01:29:13.425 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' image = image ']' 2022-04-10T01:29:13.426 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_7 snapshot 2022-04-10T01:29:13.426 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:29:13.426 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:29:13.426 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_7 2022-04-10T01:29:13.427 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:29:13.427 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:29:13.427 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:29:13.427 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:29:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:29:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2022-04-10T01:29:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:29:14.336 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2022-04-10T01:29:14.371 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_7': 2022-04-10T01:29:14.371 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:29:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:29:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:29:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 1293a10ac086 2022-04-10T01:29:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.1293a10ac086 2022-04-10T01:29:14.373 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:29:14.373 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:29:14.373 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:29:14.373 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:29:14.373 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:29:13 2022 2022-04-10T01:29:14.374 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:29:13 2022 2022-04-10T01:29:14.374 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:29:13 2022 2022-04-10T01:29:14.374 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:29:14.374 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:29:14.374 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 551577c0-7aad-4cdb-945f-ca54529923ea 2022-04-10T01:29:14.375 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:29:14.376 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:29:14.376 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_7 2022-04-10T01:29:14.377 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:29:14.377 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:29:14.377 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_7 2022-04-10T01:29:14.377 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:29:14.377 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=6 2022-04-10T01:29:14.378 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:29:14.378 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:29:14.378 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:29:14.378 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:29:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:29:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:29:14.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 106307584 bytes 2022-04-10T01:29:14.905 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 171600896 bytes 2022-04-10T01:29:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 209767424 bytes 2022-04-10T01:29:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 260665344 bytes 2022-04-10T01:29:15.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 287601664 bytes 2022-04-10T01:29:15.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 296673280 bytes 2022-04-10T01:29:15.874 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 342746112 bytes 2022-04-10T01:29:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 355189760 bytes 2022-04-10T01:29:16.245 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 392828928 bytes 2022-04-10T01:29:16.465 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 439772160 bytes 2022-04-10T01:29:16.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 500008960 bytes 2022-04-10T01:29:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 552501248 bytes 2022-04-10T01:29:17.077 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 553101312 bytes 2022-04-10T01:29:17.342 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 640375808 bytes 2022-04-10T01:29:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 674553856 bytes 2022-04-10T01:29:17.730 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 699727872 bytes 2022-04-10T01:29:17.992 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 784937984 bytes 2022-04-10T01:29:18.258 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 881467392 bytes 2022-04-10T01:29:18.486 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 940692480 bytes 2022-04-10T01:29:18.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 983066624 bytes 2022-04-10T01:29:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 1014396928 bytes 2022-04-10T01:29:19.115 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 1075185664 bytes 2022-04-10T01:29:19.322 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 1139297280 bytes 2022-04-10T01:29:19.505 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 1148772352 bytes 2022-04-10T01:29:19.731 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 1207432192 bytes 2022-04-10T01:29:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1271369728 bytes 2022-04-10T01:29:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1296225280 bytes 2022-04-10T01:29:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1298937856 bytes 2022-04-10T01:29:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:29:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:29:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:29:20.391 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:29:20.391 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:29:20.391 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_8 2022-04-10T01:29:20.391 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image_and_enable_mirror cluster2 mirror image_8 snapshot 128M 2022-04-10T01:29:20.392 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:29:20.392 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:29:20.392 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:29:20.392 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:29:20.392 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_8 2022-04-10T01:29:20.393 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:29:20.393 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:29:20.393 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:29:20.393 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:29:20.394 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:29:20.394 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:29:20.394 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snapshot ']' 2022-04-10T01:29:20.394 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:29:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_8 128M 2022-04-10T01:29:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:29:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:29:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:29:20.396 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:29:20.396 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_8 2022-04-10T01:29:20.396 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:29:20.396 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:29:20.396 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:29:20.397 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:29:20.397 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:29:20.397 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:29:20.397 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 128M ']' 2022-04-10T01:29:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=128M 2022-04-10T01:29:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:29:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_8 2022-04-10T01:29:20.441 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' image = image ']' 2022-04-10T01:29:20.442 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_8 snapshot 2022-04-10T01:29:20.442 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:29:20.442 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:29:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_8 2022-04-10T01:29:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:29:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:29:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:29:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:29:20.444 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:29:20.444 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2022-04-10T01:29:21.317 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:29:21.330 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2022-04-10T01:29:21.365 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_8': 2022-04-10T01:29:21.365 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:29:21.365 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:29:21.366 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:29:21.366 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 129db4142813 2022-04-10T01:29:21.366 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.129db4142813 2022-04-10T01:29:21.366 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:29:21.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:29:21.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:29:21.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:29:21.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:29:20 2022 2022-04-10T01:29:21.368 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:29:20 2022 2022-04-10T01:29:21.368 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:29:20 2022 2022-04-10T01:29:21.368 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:29:21.368 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:29:21.368 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 77c1be68-fcf7-4bbb-bf06-177e6f3d4112 2022-04-10T01:29:21.369 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:29:21.370 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:29:21.371 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_8 2022-04-10T01:29:21.371 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:29:21.371 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:29:21.372 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_8 2022-04-10T01:29:21.372 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:29:21.372 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=13 2022-04-10T01:29:21.372 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:29:21.372 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:29:21.373 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:29:21.373 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:29:21.373 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:29:21.373 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:29:21.592 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 36216832 bytes 2022-04-10T01:29:21.792 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 84373504 bytes 2022-04-10T01:29:22.015 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 151292928 bytes 2022-04-10T01:29:22.289 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 241642496 bytes 2022-04-10T01:29:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 273945600 bytes 2022-04-10T01:29:22.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 274557952 bytes 2022-04-10T01:29:22.812 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 298526720 bytes 2022-04-10T01:29:22.991 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr:2022-04-10T01:29:22.988+0000 7f52fc8d6700 -1 reset not still connected to 0x5600e16d9e10 2022-04-10T01:29:23.002 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 333011968 bytes 2022-04-10T01:29:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 375393280 bytes 2022-04-10T01:29:23.449 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 453601280 bytes 2022-04-10T01:29:23.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 511397888 bytes 2022-04-10T01:29:23.869 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 553952256 bytes 2022-04-10T01:29:24.119 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 632565760 bytes 2022-04-10T01:29:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 720859136 bytes 2022-04-10T01:29:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 745036800 bytes 2022-04-10T01:29:24.752 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 782940160 bytes 2022-04-10T01:29:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 833857536 bytes 2022-04-10T01:29:25.184 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 892107776 bytes 2022-04-10T01:29:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 923147264 bytes 2022-04-10T01:29:25.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 929277952 bytes 2022-04-10T01:29:25.703 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 954952704 bytes 2022-04-10T01:29:25.881 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 982065152 bytes 2022-04-10T01:29:26.089 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 1030989824 bytes 2022-04-10T01:29:26.354 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 1113558016 bytes 2022-04-10T01:29:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 1152235520 bytes 2022-04-10T01:29:26.783 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1231201280 bytes 2022-04-10T01:29:26.960 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1250921472 bytes 2022-04-10T01:29:27.202 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1325996032 bytes 2022-04-10T01:29:27.420 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1380961280 bytes 2022-04-10T01:29:27.606 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1413529600 bytes 2022-04-10T01:29:27.799 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1451533312 bytes 2022-04-10T01:29:28.038 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1508274176 bytes 2022-04-10T01:29:28.263 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1583854592 bytes 2022-04-10T01:29:28.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1634157568 bytes 2022-04-10T01:29:28.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1657288704 bytes 2022-04-10T01:29:28.860 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1709986816 bytes 2022-04-10T01:29:29.023 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1719342080 bytes 2022-04-10T01:29:29.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1749004288 bytes 2022-04-10T01:29:29.390 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1790701568 bytes 2022-04-10T01:29:29.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1887435776 bytes 2022-04-10T01:29:29.813 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2022-04-10T01:29:29.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2022-04-10T01:29:29.814 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2022-04-10T01:29:29.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi043.stderr:2022-04-10T01:29:29.812+0000 7fae5200a700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 47516) UID: 1000 2022-04-10T01:29:29.914 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1951262720 bytes 2022-04-10T01:29:30.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi043.stderr:daemon-helper: command failed with exit status 1 2022-04-10T01:29:30.088 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 1970477056 bytes 2022-04-10T01:29:30.291 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 2012316672 bytes 2022-04-10T01:29:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 2073596928 bytes 2022-04-10T01:29:30.709 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 2101244928 bytes 2022-04-10T01:29:30.930 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 2157667328 bytes 2022-04-10T01:29:31.186 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 2238477312 bytes 2022-04-10T01:29:31.369 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 2266312704 bytes 2022-04-10T01:29:31.617 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 2350133248 bytes 2022-04-10T01:29:31.839 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 2408295424 bytes 2022-04-10T01:29:32.043 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 2447709184 bytes 2022-04-10T01:29:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 2471948288 bytes 2022-04-10T01:29:32.428 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 2524084224 bytes 2022-04-10T01:29:32.601 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5400 IOs, 2537652224 bytes 2022-04-10T01:29:32.782 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5500 IOs, 2560802816 bytes 2022-04-10T01:29:32.971 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5600 IOs, 2594853888 bytes 2022-04-10T01:29:33.235 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5700 IOs, 2695025664 bytes 2022-04-10T01:29:33.465 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5800 IOs, 2751568896 bytes 2022-04-10T01:29:33.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5900 IOs, 2834990080 bytes 2022-04-10T01:29:33.815 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2022-04-10T01:29:33.816 INFO:teuthology.orchestra.run:waiting for 600 2022-04-10T01:29:33.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-04-10T01:29:33.817 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2022-04-10T01:29:33.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2022-04-10T01:29:33.818 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2022-04-10T01:29:33.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2022-04-10T01:29:33.826 INFO:tasks.rbd_mirror_thrash:waiting for 108 secs before thrashing 2022-04-10T01:29:33.928 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6000 IOs, 2857113600 bytes 2022-04-10T01:29:34.171 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6100 IOs, 2912686080 bytes 2022-04-10T01:29:34.351 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6200 IOs, 2941202432 bytes 2022-04-10T01:29:34.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:29:34.380 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:29:34.380 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:29:34.380 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:29:34.380 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:29:34.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_9 2022-04-10T01:29:34.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image_and_enable_mirror cluster2 mirror image_9 snapshot 128M 2022-04-10T01:29:34.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:29:34.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:29:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:29:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:29:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_9 2022-04-10T01:29:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:29:34.383 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:29:34.383 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:29:34.383 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:29:34.383 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:29:34.383 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:29:34.384 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snapshot ']' 2022-04-10T01:29:34.384 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:29:34.384 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_9 128M 2022-04-10T01:29:34.384 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:29:34.385 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:29:34.385 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:29:34.385 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:29:34.385 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_9 2022-04-10T01:29:34.385 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:29:34.386 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:29:34.386 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:29:34.386 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:29:34.386 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:29:34.387 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:29:34.387 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 128M ']' 2022-04-10T01:29:34.387 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=128M 2022-04-10T01:29:34.388 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:29:34.388 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_9 2022-04-10T01:29:34.431 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' image = image ']' 2022-04-10T01:29:34.431 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_9 snapshot 2022-04-10T01:29:34.431 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:29:34.432 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:29:34.432 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_9 2022-04-10T01:29:34.432 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:29:34.432 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:29:34.432 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:29:34.433 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:29:34.433 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:29:34.433 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2022-04-10T01:29:35.330 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:29:35.346 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2022-04-10T01:29:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_9': 2022-04-10T01:29:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:29:35.382 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:29:35.382 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:29:35.382 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 12aafb820470 2022-04-10T01:29:35.383 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.12aafb820470 2022-04-10T01:29:35.383 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:29:35.383 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:29:35.383 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:29:35.384 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:29:35.384 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:29:34 2022 2022-04-10T01:29:35.385 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:29:34 2022 2022-04-10T01:29:35.385 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:29:34 2022 2022-04-10T01:29:35.385 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:29:35.386 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:29:35.386 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 0661b513-be8b-4e37-9a6b-b32c1a1daf4e 2022-04-10T01:29:35.387 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:29:35.390 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:29:35.391 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_9 2022-04-10T01:29:35.391 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:29:35.391 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:29:35.392 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_9 2022-04-10T01:29:35.392 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:29:35.393 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=9 2022-04-10T01:29:35.393 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:29:35.394 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:29:35.394 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:29:35.395 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:29:35.395 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:29:35.396 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:29:35.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 71088128 bytes 2022-04-10T01:29:35.830 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 84292608 bytes 2022-04-10T01:29:36.008 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 125333504 bytes 2022-04-10T01:29:36.032 INFO:tasks.ceph.cluster2.osd.2.smithi043.stderr:2022-04-10T01:29:36.028+0000 7f8be0453700 -1 reset not still connected to 0x55f2a4636c30 2022-04-10T01:29:36.032 INFO:tasks.ceph.cluster2.osd.2.smithi043.stderr:2022-04-10T01:29:36.028+0000 7f8be0453700 -1 reset not still connected to 0x55f2a5a39110 2022-04-10T01:29:36.226 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 169624576 bytes 2022-04-10T01:29:36.391 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 178352128 bytes 2022-04-10T01:29:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 203408384 bytes 2022-04-10T01:29:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 228059136 bytes 2022-04-10T01:29:36.899 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 246129664 bytes 2022-04-10T01:29:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 279843840 bytes 2022-04-10T01:29:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 340009984 bytes 2022-04-10T01:29:37.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 353063936 bytes 2022-04-10T01:29:37.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 406398976 bytes 2022-04-10T01:29:37.896 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 464678912 bytes 2022-04-10T01:29:38.109 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 513041408 bytes 2022-04-10T01:29:38.298 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 546679808 bytes 2022-04-10T01:29:38.447 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 547301376 bytes 2022-04-10T01:29:38.632 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 579796992 bytes 2022-04-10T01:29:38.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 609453056 bytes 2022-04-10T01:29:39.009 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 655785984 bytes 2022-04-10T01:29:39.210 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 702385152 bytes 2022-04-10T01:29:39.446 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 757892096 bytes 2022-04-10T01:29:39.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 796060672 bytes 2022-04-10T01:29:39.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 821574656 bytes 2022-04-10T01:29:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 862638080 bytes 2022-04-10T01:29:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 912549888 bytes 2022-04-10T01:29:40.416 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 944334848 bytes 2022-04-10T01:29:40.625 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1000731648 bytes 2022-04-10T01:29:40.823 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1042072576 bytes 2022-04-10T01:29:41.147 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1130864640 bytes 2022-04-10T01:29:41.412 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1203686400 bytes 2022-04-10T01:29:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1204268032 bytes 2022-04-10T01:29:41.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1210268672 bytes 2022-04-10T01:29:41.868 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1210896384 bytes 2022-04-10T01:29:42.116 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1236883456 bytes 2022-04-10T01:29:42.298 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1252313088 bytes 2022-04-10T01:29:42.472 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1274769408 bytes 2022-04-10T01:29:42.698 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1349083136 bytes 2022-04-10T01:29:42.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1377802240 bytes 2022-04-10T01:29:43.086 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1423030272 bytes 2022-04-10T01:29:43.322 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1492825088 bytes 2022-04-10T01:29:43.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1496643584 bytes 2022-04-10T01:29:43.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 1539635200 bytes 2022-04-10T01:29:43.900 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 1600415744 bytes 2022-04-10T01:29:44.057 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 1601034240 bytes 2022-04-10T01:29:44.263 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 1646714880 bytes 2022-04-10T01:29:44.394 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:29:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:29:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:29:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:29:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:29:44.396 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_10 2022-04-10T01:29:44.396 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image_and_enable_mirror cluster2 mirror image_10 snapshot 128M 2022-04-10T01:29:44.396 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:29:44.396 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:29:44.397 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:29:44.397 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:29:44.397 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_10 2022-04-10T01:29:44.397 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:29:44.397 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:29:44.398 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:29:44.398 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:29:44.398 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:29:44.398 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:29:44.399 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snapshot ']' 2022-04-10T01:29:44.399 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:29:44.399 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_10 128M 2022-04-10T01:29:44.399 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:29:44.400 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:29:44.400 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:29:44.400 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:29:44.400 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_10 2022-04-10T01:29:44.401 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:29:44.401 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:29:44.401 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:29:44.401 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:29:44.402 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:29:44.402 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:29:44.402 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 128M ']' 2022-04-10T01:29:44.403 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=128M 2022-04-10T01:29:44.403 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:29:44.403 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_10 2022-04-10T01:29:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' image = image ']' 2022-04-10T01:29:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_10 snapshot 2022-04-10T01:29:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:29:44.442 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:29:44.442 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_10 2022-04-10T01:29:44.442 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:29:44.442 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:29:44.443 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:29:44.443 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:29:44.443 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:29:44.443 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2022-04-10T01:29:45.329 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:29:45.343 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2022-04-10T01:29:45.377 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_10': 2022-04-10T01:29:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:29:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:29:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:29:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 12b357b4f26a 2022-04-10T01:29:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.12b357b4f26a 2022-04-10T01:29:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:29:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:29:45.380 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:29:45.380 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:29:45.380 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:29:44 2022 2022-04-10T01:29:45.380 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:29:44 2022 2022-04-10T01:29:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:29:44 2022 2022-04-10T01:29:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:29:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:29:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: e9d2f488-240a-4db7-8e1f-b458c15d12a8 2022-04-10T01:29:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:29:45.383 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:29:45.384 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_10 2022-04-10T01:29:45.384 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:29:45.384 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:29:45.384 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_10 2022-04-10T01:29:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:29:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=9 2022-04-10T01:29:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:29:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:29:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:29:45.386 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:29:45.386 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:29:45.386 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:29:45.587 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 9112576 bytes 2022-04-10T01:29:45.779 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 46828544 bytes 2022-04-10T01:29:45.997 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 90370048 bytes 2022-04-10T01:29:46.195 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 132463616 bytes 2022-04-10T01:29:46.455 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 206623744 bytes 2022-04-10T01:29:46.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 262084608 bytes 2022-04-10T01:29:46.852 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 295921664 bytes 2022-04-10T01:29:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 303653888 bytes 2022-04-10T01:29:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 329935872 bytes 2022-04-10T01:29:47.435 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 386249728 bytes 2022-04-10T01:29:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 460053504 bytes 2022-04-10T01:29:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 491232256 bytes 2022-04-10T01:29:48.048 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 491964416 bytes 2022-04-10T01:29:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 531619840 bytes 2022-04-10T01:29:48.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 572482560 bytes 2022-04-10T01:29:48.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 605667328 bytes 2022-04-10T01:29:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 627876864 bytes 2022-04-10T01:29:49.038 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 653713408 bytes 2022-04-10T01:29:49.225 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 665380864 bytes 2022-04-10T01:29:49.440 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 716513280 bytes 2022-04-10T01:29:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 779950080 bytes 2022-04-10T01:29:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 838977536 bytes 2022-04-10T01:29:50.124 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 883676160 bytes 2022-04-10T01:29:50.295 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 893433856 bytes 2022-04-10T01:29:50.468 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 899672064 bytes 2022-04-10T01:29:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 934108160 bytes 2022-04-10T01:29:50.920 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1009620992 bytes 2022-04-10T01:29:51.096 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1010254848 bytes 2022-04-10T01:29:51.326 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1070367744 bytes 2022-04-10T01:29:51.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1116276736 bytes 2022-04-10T01:29:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1211973632 bytes 2022-04-10T01:29:52.010 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1252143104 bytes 2022-04-10T01:29:52.225 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1300886528 bytes 2022-04-10T01:29:52.435 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1339318272 bytes 2022-04-10T01:29:52.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1381691392 bytes 2022-04-10T01:29:52.847 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1411180544 bytes 2022-04-10T01:29:53.018 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1430334464 bytes 2022-04-10T01:29:53.188 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1461728256 bytes 2022-04-10T01:29:53.377 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1462738944 bytes 2022-04-10T01:29:53.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1551399936 bytes 2022-04-10T01:29:53.855 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1587209216 bytes 2022-04-10T01:29:54.033 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 1631810560 bytes 2022-04-10T01:29:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 1691735040 bytes 2022-04-10T01:29:54.377 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 1736083456 bytes 2022-04-10T01:29:54.392 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:29:54.392 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:29:54.392 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:29:54.393 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:29:54.393 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:29:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_11 2022-04-10T01:29:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image_and_enable_mirror cluster2 mirror image_11 snapshot 128M 2022-04-10T01:29:54.395 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:29:54.395 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:29:54.396 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:29:54.396 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:29:54.396 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_11 2022-04-10T01:29:54.397 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:29:54.397 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:29:54.398 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:29:54.398 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:29:54.399 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:29:54.399 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:29:54.400 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snapshot ']' 2022-04-10T01:29:54.400 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:29:54.401 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_11 128M 2022-04-10T01:29:54.401 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:29:54.401 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:29:54.402 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:29:54.402 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:29:54.403 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_11 2022-04-10T01:29:54.403 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:29:54.404 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:29:54.404 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:29:54.405 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:29:54.405 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:29:54.405 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:29:54.406 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 128M ']' 2022-04-10T01:29:54.406 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=128M 2022-04-10T01:29:54.407 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:29:54.407 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_11 2022-04-10T01:29:54.448 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' image = image ']' 2022-04-10T01:29:54.448 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_11 snapshot 2022-04-10T01:29:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:29:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:29:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_11 2022-04-10T01:29:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:29:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:29:54.450 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:29:54.450 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:29:54.450 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:29:54.450 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_11 snapshot 2022-04-10T01:29:55.316 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:29:55.324 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_11 2022-04-10T01:29:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_11': 2022-04-10T01:29:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:29:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:29:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:29:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 12bc4feb24dd 2022-04-10T01:29:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.12bc4feb24dd 2022-04-10T01:29:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:29:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:29:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:29:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:29:55.364 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:29:54 2022 2022-04-10T01:29:55.364 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:29:54 2022 2022-04-10T01:29:55.364 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:29:54 2022 2022-04-10T01:29:55.364 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:29:55.364 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:29:55.365 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: ff2a6efd-2c82-4070-8950-c9c7f1385660 2022-04-10T01:29:55.365 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:29:55.366 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:29:55.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_11 2022-04-10T01:29:55.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:29:55.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:29:55.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_11 2022-04-10T01:29:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:29:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=9 2022-04-10T01:29:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:29:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:29:55.369 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:29:55.369 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:29:55.369 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:29:55.369 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_11 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:29:55.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 68606976 bytes 2022-04-10T01:29:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 100356096 bytes 2022-04-10T01:29:55.862 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 138610688 bytes 2022-04-10T01:29:55.971 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 149070848 bytes 2022-04-10T01:29:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 178487296 bytes 2022-04-10T01:29:56.216 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 198671360 bytes 2022-04-10T01:29:56.318 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 204516352 bytes 2022-04-10T01:29:56.451 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 232091648 bytes 2022-04-10T01:29:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 343877632 bytes 2022-04-10T01:29:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 453488640 bytes 2022-04-10T01:29:57.012 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 454090752 bytes 2022-04-10T01:29:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 496930816 bytes 2022-04-10T01:29:57.278 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 514922496 bytes 2022-04-10T01:29:57.448 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 567684096 bytes 2022-04-10T01:29:57.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 570283008 bytes 2022-04-10T01:29:57.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 602119168 bytes 2022-04-10T01:29:57.809 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 634917888 bytes 2022-04-10T01:29:57.994 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 693055488 bytes 2022-04-10T01:29:58.163 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 748061696 bytes 2022-04-10T01:29:58.384 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 831637504 bytes 2022-04-10T01:29:58.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 893714432 bytes 2022-04-10T01:29:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 947780608 bytes 2022-04-10T01:29:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 988473344 bytes 2022-04-10T01:29:59.032 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 1031496704 bytes 2022-04-10T01:29:59.152 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 1051956224 bytes 2022-04-10T01:29:59.272 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1071171584 bytes 2022-04-10T01:29:59.434 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1122488320 bytes 2022-04-10T01:29:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1181216768 bytes 2022-04-10T01:29:59.791 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1238757376 bytes 2022-04-10T01:29:59.908 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1266592768 bytes 2022-04-10T01:30:00.059 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1289910272 bytes 2022-04-10T01:30:00.273 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1379376128 bytes 2022-04-10T01:30:00.440 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1438985216 bytes 2022-04-10T01:30:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1505435648 bytes 2022-04-10T01:30:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1536143360 bytes 2022-04-10T01:30:00.924 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1562695680 bytes 2022-04-10T01:30:01.057 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1594701824 bytes 2022-04-10T01:30:01.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1631893504 bytes 2022-04-10T01:30:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1697423360 bytes 2022-04-10T01:30:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1781537792 bytes 2022-04-10T01:30:01.718 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1811190784 bytes 2022-04-10T01:30:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 1860210688 bytes 2022-04-10T01:30:02.035 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 1891414016 bytes 2022-04-10T01:30:02.136 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 1892032512 bytes 2022-04-10T01:30:02.266 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 1911992320 bytes 2022-04-10T01:30:02.457 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 1992214528 bytes 2022-04-10T01:30:02.615 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 2038153216 bytes 2022-04-10T01:30:02.759 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 2093913088 bytes 2022-04-10T01:30:02.967 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 2167810048 bytes 2022-04-10T01:30:03.094 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 2169194496 bytes 2022-04-10T01:30:03.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 2204406784 bytes 2022-04-10T01:30:03.406 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 2249100288 bytes 2022-04-10T01:30:03.544 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 2295100416 bytes 2022-04-10T01:30:03.730 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5400 IOs, 2344497152 bytes 2022-04-10T01:30:03.903 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5500 IOs, 2407534592 bytes 2022-04-10T01:30:04.056 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5600 IOs, 2444455936 bytes 2022-04-10T01:30:04.211 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5700 IOs, 2492579840 bytes 2022-04-10T01:30:04.341 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5800 IOs, 2510964736 bytes 2022-04-10T01:30:04.373 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:30:04.373 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:30:04.373 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:30:04.373 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:30:04.374 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:30:04.374 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_12 2022-04-10T01:30:04.374 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image_and_enable_mirror cluster2 mirror image_12 snapshot 128M 2022-04-10T01:30:04.374 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:30:04.375 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:30:04.375 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:30:04.375 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:30:04.375 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_12 2022-04-10T01:30:04.375 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:30:04.378 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:30:04.378 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:30:04.378 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:30:04.378 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:30:04.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:30:04.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snapshot ']' 2022-04-10T01:30:04.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:30:04.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_12 128M 2022-04-10T01:30:04.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:30:04.380 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:30:04.380 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:30:04.380 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:30:04.380 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_12 2022-04-10T01:30:04.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:30:04.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:30:04.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:30:04.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:30:04.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:30:04.382 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:30:04.382 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 128M ']' 2022-04-10T01:30:04.382 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=128M 2022-04-10T01:30:04.383 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:30:04.383 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_12 2022-04-10T01:30:04.417 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' image = image ']' 2022-04-10T01:30:04.417 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_12 snapshot 2022-04-10T01:30:04.417 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:30:04.417 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:30:04.418 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_12 2022-04-10T01:30:04.418 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:30:04.418 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:30:04.419 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:30:04.419 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:30:04.419 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:30:04.420 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_12 snapshot 2022-04-10T01:30:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:30:05.340 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_12 2022-04-10T01:30:05.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_12': 2022-04-10T01:30:05.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:30:05.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:30:05.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:30:05.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 12c531d8c65e 2022-04-10T01:30:05.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.12c531d8c65e 2022-04-10T01:30:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:30:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:30:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:30:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:30:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:30:04 2022 2022-04-10T01:30:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:30:04 2022 2022-04-10T01:30:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:30:04 2022 2022-04-10T01:30:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:30:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:30:05.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 672aa5fe-c483-4bea-94d8-2b504c745764 2022-04-10T01:30:05.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:30:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:30:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_12 2022-04-10T01:30:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:30:05.583 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:30:05.583 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_12 2022-04-10T01:30:05.583 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:30:05.584 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=12 2022-04-10T01:30:05.584 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:30:05.585 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:30:05.585 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:30:05.585 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:30:05.585 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:30:05.586 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_12 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:30:05.824 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 66029568 bytes 2022-04-10T01:30:05.945 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 86774784 bytes 2022-04-10T01:30:06.127 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 155309056 bytes 2022-04-10T01:30:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 223375360 bytes 2022-04-10T01:30:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 234202112 bytes 2022-04-10T01:30:06.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 272613376 bytes 2022-04-10T01:30:06.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 299268096 bytes 2022-04-10T01:30:06.851 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 322835456 bytes 2022-04-10T01:30:06.974 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 336525312 bytes 2022-04-10T01:30:07.120 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 363119616 bytes 2022-04-10T01:30:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 410000384 bytes 2022-04-10T01:30:07.443 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 447739904 bytes 2022-04-10T01:30:07.604 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 496429056 bytes 2022-04-10T01:30:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 538990592 bytes 2022-04-10T01:30:07.916 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 571289600 bytes 2022-04-10T01:30:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 580623360 bytes 2022-04-10T01:30:08.154 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 610471936 bytes 2022-04-10T01:30:08.375 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 673425408 bytes 2022-04-10T01:30:08.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 722130944 bytes 2022-04-10T01:30:08.787 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 817690624 bytes 2022-04-10T01:30:09.063 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 953586688 bytes 2022-04-10T01:30:09.269 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 1046329344 bytes 2022-04-10T01:30:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 1069516800 bytes 2022-04-10T01:30:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 1117034496 bytes 2022-04-10T01:30:09.757 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 1176270848 bytes 2022-04-10T01:30:09.879 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1176863744 bytes 2022-04-10T01:30:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1190251520 bytes 2022-04-10T01:30:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1249072128 bytes 2022-04-10T01:30:10.302 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1266191360 bytes 2022-04-10T01:30:10.451 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1284660224 bytes 2022-04-10T01:30:10.609 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1325162496 bytes 2022-04-10T01:30:10.838 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1396105216 bytes 2022-04-10T01:30:11.027 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1458790400 bytes 2022-04-10T01:30:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1510278144 bytes 2022-04-10T01:30:11.336 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1534317568 bytes 2022-04-10T01:30:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1640936448 bytes 2022-04-10T01:30:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1641737216 bytes 2022-04-10T01:30:11.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1649433600 bytes 2022-04-10T01:30:11.948 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1664062464 bytes 2022-04-10T01:30:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1704772608 bytes 2022-04-10T01:30:12.247 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1727298560 bytes 2022-04-10T01:30:12.462 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 1810126848 bytes 2022-04-10T01:30:12.625 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 1852240896 bytes 2022-04-10T01:30:12.760 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 1885607936 bytes 2022-04-10T01:30:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 1900653568 bytes 2022-04-10T01:30:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 1959265280 bytes 2022-04-10T01:30:13.230 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 2002732032 bytes 2022-04-10T01:30:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 2025799680 bytes 2022-04-10T01:30:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 2041808896 bytes 2022-04-10T01:30:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 2080434176 bytes 2022-04-10T01:30:13.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 2161153024 bytes 2022-04-10T01:30:14.039 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 2204609536 bytes 2022-04-10T01:30:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 2299032576 bytes 2022-04-10T01:30:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5400 IOs, 2333521920 bytes 2022-04-10T01:30:14.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5500 IOs, 2353733632 bytes 2022-04-10T01:30:14.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5600 IOs, 2377302016 bytes 2022-04-10T01:30:14.825 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5700 IOs, 2409289728 bytes 2022-04-10T01:30:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5800 IOs, 2485041152 bytes 2022-04-10T01:30:15.198 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5900 IOs, 2536344576 bytes 2022-04-10T01:30:15.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6000 IOs, 2569786368 bytes 2022-04-10T01:30:15.541 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6100 IOs, 2635296768 bytes 2022-04-10T01:30:15.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6200 IOs, 2681168896 bytes 2022-04-10T01:30:15.855 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6300 IOs, 2714914816 bytes 2022-04-10T01:30:15.970 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6400 IOs, 2721996800 bytes 2022-04-10T01:30:16.087 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6500 IOs, 2731794432 bytes 2022-04-10T01:30:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6600 IOs, 2732407808 bytes 2022-04-10T01:30:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6700 IOs, 2751423488 bytes 2022-04-10T01:30:16.489 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6800 IOs, 2789430272 bytes 2022-04-10T01:30:16.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6900 IOs, 2823796736 bytes 2022-04-10T01:30:16.837 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7000 IOs, 2908612608 bytes 2022-04-10T01:30:17.090 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7100 IOs, 3019479040 bytes 2022-04-10T01:30:17.262 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7200 IOs, 3070837760 bytes 2022-04-10T01:30:17.411 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7300 IOs, 3101157376 bytes 2022-04-10T01:30:17.590 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:30:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:30:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:30:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:30:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:30:17.592 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_13 2022-04-10T01:30:17.592 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image_and_enable_mirror cluster2 mirror image_13 snapshot 128M 2022-04-10T01:30:17.592 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:30:17.592 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:30:17.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:30:17.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:30:17.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_13 2022-04-10T01:30:17.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:30:17.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:30:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:30:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:30:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:30:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:30:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snapshot ']' 2022-04-10T01:30:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:30:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_13 128M 2022-04-10T01:30:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:30:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:30:17.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:30:17.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:30:17.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_13 2022-04-10T01:30:17.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:30:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:30:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:30:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:30:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:30:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:30:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 128M ']' 2022-04-10T01:30:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=128M 2022-04-10T01:30:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:30:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_13 2022-04-10T01:30:17.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' image = image ']' 2022-04-10T01:30:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_13 snapshot 2022-04-10T01:30:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:30:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:30:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_13 2022-04-10T01:30:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:30:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:30:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:30:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:30:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:30:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_13 snapshot 2022-04-10T01:30:18.378 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:30:18.390 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_13 2022-04-10T01:30:18.426 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_13': 2022-04-10T01:30:18.427 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:30:18.427 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:30:18.427 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:30:18.428 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 12cea65635a5 2022-04-10T01:30:18.428 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.12cea65635a5 2022-04-10T01:30:18.429 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:30:18.429 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:30:18.430 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:30:18.430 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:30:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:30:17 2022 2022-04-10T01:30:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:30:17 2022 2022-04-10T01:30:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:30:17 2022 2022-04-10T01:30:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:30:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:30:18.433 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: b7fe287d-2790-4653-8a15-70f6e8da8e9f 2022-04-10T01:30:18.433 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:30:18.434 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:30:18.435 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_13 2022-04-10T01:30:18.435 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:30:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:30:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_13 2022-04-10T01:30:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:30:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=10 2022-04-10T01:30:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:30:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:30:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:30:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:30:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:30:18.438 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_13 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:30:18.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 70645760 bytes 2022-04-10T01:30:18.795 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 89256960 bytes 2022-04-10T01:30:18.937 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 123651072 bytes 2022-04-10T01:30:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 184690688 bytes 2022-04-10T01:30:19.269 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 228133888 bytes 2022-04-10T01:30:19.456 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 296348672 bytes 2022-04-10T01:30:19.607 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 357166080 bytes 2022-04-10T01:30:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 360904704 bytes 2022-04-10T01:30:19.873 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 378250240 bytes 2022-04-10T01:30:20.005 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 397863936 bytes 2022-04-10T01:30:20.170 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 444280832 bytes 2022-04-10T01:30:20.325 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 479516672 bytes 2022-04-10T01:30:20.454 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 497388544 bytes 2022-04-10T01:30:20.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 586053632 bytes 2022-04-10T01:30:20.819 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 608873472 bytes 2022-04-10T01:30:20.959 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 634722304 bytes 2022-04-10T01:30:21.126 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 684869632 bytes 2022-04-10T01:30:21.310 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 747784192 bytes 2022-04-10T01:30:21.467 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 794388480 bytes 2022-04-10T01:30:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 832379904 bytes 2022-04-10T01:30:21.750 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 848679936 bytes 2022-04-10T01:30:21.909 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 884636672 bytes 2022-04-10T01:30:22.031 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 887377920 bytes 2022-04-10T01:30:22.137 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 913386496 bytes 2022-04-10T01:30:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 994441216 bytes 2022-04-10T01:30:22.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1046045696 bytes 2022-04-10T01:30:22.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1081489408 bytes 2022-04-10T01:30:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1164197888 bytes 2022-04-10T01:30:22.972 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1189013504 bytes 2022-04-10T01:30:23.103 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1237505024 bytes 2022-04-10T01:30:23.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1265081344 bytes 2022-04-10T01:30:23.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1308026880 bytes 2022-04-10T01:30:23.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1342620672 bytes 2022-04-10T01:30:23.628 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1369076736 bytes 2022-04-10T01:30:23.743 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1396022272 bytes 2022-04-10T01:30:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1423520768 bytes 2022-04-10T01:30:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1462606848 bytes 2022-04-10T01:30:24.110 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1463226368 bytes 2022-04-10T01:30:24.267 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1527796736 bytes 2022-04-10T01:30:24.419 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1575734272 bytes 2022-04-10T01:30:24.543 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1606381568 bytes 2022-04-10T01:30:24.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 1623038976 bytes 2022-04-10T01:30:24.765 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 1644894208 bytes 2022-04-10T01:30:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 1730907136 bytes 2022-04-10T01:30:25.134 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 1823310848 bytes 2022-04-10T01:30:25.275 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 1853693952 bytes 2022-04-10T01:30:25.395 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 1894966272 bytes 2022-04-10T01:30:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 1984775168 bytes 2022-04-10T01:30:25.702 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 1985387520 bytes 2022-04-10T01:30:25.866 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 2044828672 bytes 2022-04-10T01:30:26.048 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 2117605376 bytes 2022-04-10T01:30:26.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 2171087872 bytes 2022-04-10T01:30:26.416 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 2274932736 bytes 2022-04-10T01:30:26.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5400 IOs, 2338638848 bytes 2022-04-10T01:30:26.722 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5500 IOs, 2379451392 bytes 2022-04-10T01:30:26.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5600 IOs, 2385685504 bytes 2022-04-10T01:30:26.908 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5700 IOs, 2386286592 bytes 2022-04-10T01:30:27.033 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5800 IOs, 2414459904 bytes 2022-04-10T01:30:27.144 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5900 IOs, 2452669440 bytes 2022-04-10T01:30:27.298 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6000 IOs, 2486206464 bytes 2022-04-10T01:30:27.452 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6100 IOs, 2544063488 bytes 2022-04-10T01:30:27.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6200 IOs, 2559603712 bytes 2022-04-10T01:30:27.717 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6300 IOs, 2611272704 bytes 2022-04-10T01:30:27.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6400 IOs, 2682514432 bytes 2022-04-10T01:30:28.094 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6500 IOs, 2757295104 bytes 2022-04-10T01:30:28.181 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6600 IOs, 2757907456 bytes 2022-04-10T01:30:28.329 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6700 IOs, 2804480000 bytes 2022-04-10T01:30:28.420 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6800 IOs, 2805100544 bytes 2022-04-10T01:30:28.438 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:30:28.438 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:30:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:30:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:30:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:30:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_14 2022-04-10T01:30:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image_and_enable_mirror cluster2 mirror image_14 snapshot 128M 2022-04-10T01:30:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:30:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:30:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:30:28.441 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:30:28.441 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_14 2022-04-10T01:30:28.441 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:30:28.441 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:30:28.442 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:30:28.442 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:30:28.442 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:30:28.442 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:30:28.442 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snapshot ']' 2022-04-10T01:30:28.443 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:30:28.443 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_14 128M 2022-04-10T01:30:28.443 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:30:28.443 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:30:28.444 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:30:28.444 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:30:28.444 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_14 2022-04-10T01:30:28.444 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:30:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:30:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:30:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:30:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:30:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:30:28.446 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 128M ']' 2022-04-10T01:30:28.446 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=128M 2022-04-10T01:30:28.446 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:30:28.446 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_14 2022-04-10T01:30:28.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' image = image ']' 2022-04-10T01:30:28.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_14 snapshot 2022-04-10T01:30:28.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:30:28.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:30:28.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_14 2022-04-10T01:30:28.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:30:28.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:30:28.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:30:28.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:30:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:30:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_14 snapshot 2022-04-10T01:30:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:30:28.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_14 2022-04-10T01:30:28.629 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_14': 2022-04-10T01:30:28.630 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:30:28.630 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:30:28.631 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:30:28.631 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 12d6689584a 2022-04-10T01:30:28.631 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.12d6689584a 2022-04-10T01:30:28.631 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:30:28.632 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:30:28.632 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:30:28.632 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:30:28.632 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:30:28 2022 2022-04-10T01:30:28.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:30:28 2022 2022-04-10T01:30:28.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:30:28 2022 2022-04-10T01:30:28.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:30:28.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:30:28.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 9daee277-0f48-4a3a-be63-d5fc81096bc5 2022-04-10T01:30:28.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:30:28.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:30:28.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_14 2022-04-10T01:30:28.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:30:28.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:30:28.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_14 2022-04-10T01:30:28.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:30:28.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=9 2022-04-10T01:30:28.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:30:28.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:30:28.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:30:28.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:30:28.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:30:28.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_14 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:30:28.788 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 41362432 bytes 2022-04-10T01:30:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 70000640 bytes 2022-04-10T01:30:29.123 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 146789376 bytes 2022-04-10T01:30:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 189565952 bytes 2022-04-10T01:30:29.445 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 262187008 bytes 2022-04-10T01:30:29.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 300339200 bytes 2022-04-10T01:30:29.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 411831296 bytes 2022-04-10T01:30:29.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 439949312 bytes 2022-04-10T01:30:30.042 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 467068928 bytes 2022-04-10T01:30:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 513274880 bytes 2022-04-10T01:30:30.461 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 645808128 bytes 2022-04-10T01:30:30.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 705324032 bytes 2022-04-10T01:30:30.740 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 713399296 bytes 2022-04-10T01:30:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 762124288 bytes 2022-04-10T01:30:31.048 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 814077952 bytes 2022-04-10T01:30:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 881725440 bytes 2022-04-10T01:30:31.348 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 903636992 bytes 2022-04-10T01:30:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 924386304 bytes 2022-04-10T01:30:31.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 990236672 bytes 2022-04-10T01:30:31.809 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 1049931776 bytes 2022-04-10T01:30:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 1137843200 bytes 2022-04-10T01:30:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 1157673984 bytes 2022-04-10T01:30:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 1199754240 bytes 2022-04-10T01:30:32.410 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 1247355904 bytes 2022-04-10T01:30:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 1288357888 bytes 2022-04-10T01:30:32.681 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1305262080 bytes 2022-04-10T01:30:32.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1329668096 bytes 2022-04-10T01:30:32.983 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1388825600 bytes 2022-04-10T01:30:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1447956480 bytes 2022-04-10T01:30:33.333 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1507167232 bytes 2022-04-10T01:30:33.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1576560640 bytes 2022-04-10T01:30:33.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1598073856 bytes 2022-04-10T01:30:33.826 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1682213888 bytes 2022-04-10T01:30:33.966 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1725789184 bytes 2022-04-10T01:30:34.180 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1808806912 bytes 2022-04-10T01:30:34.298 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1853352960 bytes 2022-04-10T01:30:34.444 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1874394112 bytes 2022-04-10T01:30:34.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1900628992 bytes 2022-04-10T01:30:34.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1921847296 bytes 2022-04-10T01:30:34.850 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1973495808 bytes 2022-04-10T01:30:34.974 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1997546496 bytes 2022-04-10T01:30:35.148 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 2059931648 bytes 2022-04-10T01:30:35.283 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 2091627520 bytes 2022-04-10T01:30:35.449 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 2156674048 bytes 2022-04-10T01:30:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 2184269824 bytes 2022-04-10T01:30:35.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 2184884224 bytes 2022-04-10T01:30:35.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 2211298304 bytes 2022-04-10T01:30:35.903 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 2211910656 bytes 2022-04-10T01:30:36.060 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 2252557312 bytes 2022-04-10T01:30:36.177 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 2271204352 bytes 2022-04-10T01:30:36.330 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 2317685760 bytes 2022-04-10T01:30:36.472 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 2346023936 bytes 2022-04-10T01:30:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 2391558144 bytes 2022-04-10T01:30:36.841 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5400 IOs, 2496522240 bytes 2022-04-10T01:30:36.993 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5500 IOs, 2535693312 bytes 2022-04-10T01:30:37.141 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5600 IOs, 2585793536 bytes 2022-04-10T01:30:37.295 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5700 IOs, 2616635392 bytes 2022-04-10T01:30:37.440 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5800 IOs, 2655117312 bytes 2022-04-10T01:30:37.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:30:37.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:30:37.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:30:37.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:30:37.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:30:37.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_15 2022-04-10T01:30:37.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image_and_enable_mirror cluster2 mirror image_15 snapshot 128M 2022-04-10T01:30:37.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:30:37.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:30:37.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:30:37.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:30:37.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_15 2022-04-10T01:30:37.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:30:37.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:30:37.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:30:37.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:30:37.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:30:37.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:30:37.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snapshot ']' 2022-04-10T01:30:37.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:30:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_15 128M 2022-04-10T01:30:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:30:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:30:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:30:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:30:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_15 2022-04-10T01:30:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:30:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:30:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:30:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:30:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:30:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:30:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 128M ']' 2022-04-10T01:30:37.650 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=128M 2022-04-10T01:30:37.650 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:30:37.650 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_15 2022-04-10T01:30:37.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' image = image ']' 2022-04-10T01:30:37.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_15 snapshot 2022-04-10T01:30:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:30:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:30:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_15 2022-04-10T01:30:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:30:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:30:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:30:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:30:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:30:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_15 snapshot 2022-04-10T01:30:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:30:38.384 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_15 2022-04-10T01:30:38.420 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_15': 2022-04-10T01:30:38.421 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:30:38.421 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:30:38.421 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:30:38.421 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 12e0e0a206be 2022-04-10T01:30:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.12e0e0a206be 2022-04-10T01:30:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:30:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:30:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:30:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:30:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:30:37 2022 2022-04-10T01:30:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:30:37 2022 2022-04-10T01:30:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:30:37 2022 2022-04-10T01:30:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:30:38.424 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:30:38.424 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 1de7d6c4-ab1c-49a3-8053-b58d65c37430 2022-04-10T01:30:38.424 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:30:38.425 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:30:38.425 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_15 2022-04-10T01:30:38.426 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:30:38.426 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:30:38.426 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_15 2022-04-10T01:30:38.426 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:30:38.428 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=13 2022-04-10T01:30:38.428 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:30:38.428 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:30:38.428 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:30:38.429 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:30:38.429 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:30:38.429 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_15 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:30:38.660 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 83150848 bytes 2022-04-10T01:30:38.840 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 149309440 bytes 2022-04-10T01:30:39.035 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 227589120 bytes 2022-04-10T01:30:39.159 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 251826176 bytes 2022-04-10T01:30:39.268 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 264018944 bytes 2022-04-10T01:30:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 283756544 bytes 2022-04-10T01:30:39.489 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 293174272 bytes 2022-04-10T01:30:39.602 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 312442880 bytes 2022-04-10T01:30:39.778 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 384866304 bytes 2022-04-10T01:30:40.012 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 483023872 bytes 2022-04-10T01:30:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 523363328 bytes 2022-04-10T01:30:40.360 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 568024064 bytes 2022-04-10T01:30:40.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 654881792 bytes 2022-04-10T01:30:40.786 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 706201600 bytes 2022-04-10T01:30:40.948 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 763578368 bytes 2022-04-10T01:30:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 793351168 bytes 2022-04-10T01:30:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 835430400 bytes 2022-04-10T01:30:41.420 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 944887808 bytes 2022-04-10T01:30:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 1033147392 bytes 2022-04-10T01:30:41.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 1078505472 bytes 2022-04-10T01:30:41.967 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 1130192896 bytes 2022-04-10T01:30:42.062 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 1130819584 bytes 2022-04-10T01:30:42.248 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 1199064064 bytes 2022-04-10T01:30:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 1281389568 bytes 2022-04-10T01:30:42.603 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 1336186880 bytes 2022-04-10T01:30:42.773 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1404381184 bytes 2022-04-10T01:30:42.899 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1430435840 bytes 2022-04-10T01:30:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1435175936 bytes 2022-04-10T01:30:43.089 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1435795456 bytes 2022-04-10T01:30:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1510971392 bytes 2022-04-10T01:30:43.431 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1562396672 bytes 2022-04-10T01:30:43.585 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1610876928 bytes 2022-04-10T01:30:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1680351232 bytes 2022-04-10T01:30:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1727759360 bytes 2022-04-10T01:30:44.025 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1741619200 bytes 2022-04-10T01:30:44.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1816153088 bytes 2022-04-10T01:30:44.328 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1829332992 bytes 2022-04-10T01:30:44.444 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1851522048 bytes 2022-04-10T01:30:44.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1912309760 bytes 2022-04-10T01:30:44.731 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1933615104 bytes 2022-04-10T01:30:44.869 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1967841280 bytes 2022-04-10T01:30:44.999 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 1999166464 bytes 2022-04-10T01:30:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 2045605888 bytes 2022-04-10T01:30:45.289 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 2075298816 bytes 2022-04-10T01:30:45.415 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 2100183040 bytes 2022-04-10T01:30:45.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 2134195200 bytes 2022-04-10T01:30:45.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 2168509440 bytes 2022-04-10T01:30:45.768 INFO:tasks.ceph.cluster2.osd.2.smithi043.stderr:2022-04-10T01:30:45.762+0000 7f8be0453700 -1 reset not still connected to 0x55f2a45bf110 2022-04-10T01:30:45.796 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 2176730112 bytes 2022-04-10T01:30:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 2257043456 bytes 2022-04-10T01:30:46.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 2348651520 bytes 2022-04-10T01:30:46.368 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 2407105536 bytes 2022-04-10T01:30:46.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 2455519232 bytes 2022-04-10T01:30:46.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 2515497984 bytes 2022-04-10T01:30:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5400 IOs, 2543869952 bytes 2022-04-10T01:30:46.919 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5500 IOs, 2544483328 bytes 2022-04-10T01:30:47.054 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5600 IOs, 2579197952 bytes 2022-04-10T01:30:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5700 IOs, 2599647232 bytes 2022-04-10T01:30:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5800 IOs, 2614151168 bytes 2022-04-10T01:30:47.399 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5900 IOs, 2621608960 bytes 2022-04-10T01:30:47.567 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6000 IOs, 2688405504 bytes 2022-04-10T01:30:47.718 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6100 IOs, 2735702016 bytes 2022-04-10T01:30:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6200 IOs, 2754836480 bytes 2022-04-10T01:30:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6300 IOs, 2834354176 bytes 2022-04-10T01:30:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6400 IOs, 2846760960 bytes 2022-04-10T01:30:48.254 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6500 IOs, 2864258048 bytes 2022-04-10T01:30:48.396 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6600 IOs, 2899049472 bytes 2022-04-10T01:30:48.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6700 IOs, 2951920640 bytes 2022-04-10T01:30:48.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6800 IOs, 2959988736 bytes 2022-04-10T01:30:48.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6900 IOs, 3002200064 bytes 2022-04-10T01:30:48.927 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7000 IOs, 3029859328 bytes 2022-04-10T01:30:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7100 IOs, 3064522752 bytes 2022-04-10T01:30:49.258 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7200 IOs, 3153988608 bytes 2022-04-10T01:30:49.431 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7300 IOs, 3210594304 bytes 2022-04-10T01:30:49.529 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7400 IOs, 3215757312 bytes 2022-04-10T01:30:49.650 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7500 IOs, 3262025728 bytes 2022-04-10T01:30:49.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7600 IOs, 3303909376 bytes 2022-04-10T01:30:49.958 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7700 IOs, 3333302272 bytes 2022-04-10T01:30:50.121 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7800 IOs, 3380771840 bytes 2022-04-10T01:30:50.302 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7900 IOs, 3444648960 bytes 2022-04-10T01:30:50.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8000 IOs, 3508286464 bytes 2022-04-10T01:30:50.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8100 IOs, 3570480128 bytes 2022-04-10T01:30:50.766 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8200 IOs, 3591098368 bytes 2022-04-10T01:30:50.921 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8300 IOs, 3640885248 bytes 2022-04-10T01:30:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8400 IOs, 3673389056 bytes 2022-04-10T01:30:51.231 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8500 IOs, 3743817728 bytes 2022-04-10T01:30:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8600 IOs, 3757645824 bytes 2022-04-10T01:30:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:30:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:30:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:30:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:30:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:30:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_16 2022-04-10T01:30:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image_and_enable_mirror cluster2 mirror image_16 snapshot 128M 2022-04-10T01:30:51.437 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:30:51.437 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:30:51.437 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:30:51.437 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:30:51.437 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_16 2022-04-10T01:30:51.438 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:30:51.438 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:30:51.438 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:30:51.438 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:30:51.439 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:30:51.439 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:30:51.439 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snapshot ']' 2022-04-10T01:30:51.439 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:30:51.440 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_16 128M 2022-04-10T01:30:51.440 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:30:51.440 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:30:51.440 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:30:51.440 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:30:51.441 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_16 2022-04-10T01:30:51.441 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:30:51.441 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:30:51.441 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:30:51.442 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:30:51.442 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:30:51.442 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:30:51.442 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 128M ']' 2022-04-10T01:30:51.443 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=128M 2022-04-10T01:30:51.443 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:30:51.443 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_16 2022-04-10T01:30:51.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' image = image ']' 2022-04-10T01:30:51.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_16 snapshot 2022-04-10T01:30:51.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:30:51.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:30:51.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_16 2022-04-10T01:30:51.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:30:51.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:30:51.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:30:51.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:30:51.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:30:51.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_16 snapshot 2022-04-10T01:30:52.244 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:30:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_16 2022-04-10T01:30:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_16': 2022-04-10T01:30:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:30:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:30:52.292 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:30:52.292 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 12e9fb0ba25b 2022-04-10T01:30:52.292 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.12e9fb0ba25b 2022-04-10T01:30:52.292 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:30:52.293 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:30:52.293 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:30:52.293 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:30:52.293 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:30:51 2022 2022-04-10T01:30:52.293 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:30:51 2022 2022-04-10T01:30:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:30:51 2022 2022-04-10T01:30:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:30:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:30:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 47fb89d8-4dfe-49df-833b-c0b0fd34dfb9 2022-04-10T01:30:52.295 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:30:52.295 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:30:52.296 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_16 2022-04-10T01:30:52.296 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:30:52.296 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:30:52.296 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_16 2022-04-10T01:30:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:30:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=6 2022-04-10T01:30:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:30:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:30:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:30:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:30:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:30:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_16 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:30:52.455 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 17573888 bytes 2022-04-10T01:30:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 59184128 bytes 2022-04-10T01:30:52.751 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 117084160 bytes 2022-04-10T01:30:52.885 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 153711616 bytes 2022-04-10T01:30:53.036 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 199868416 bytes 2022-04-10T01:30:53.162 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 228334592 bytes 2022-04-10T01:30:53.332 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 292306944 bytes 2022-04-10T01:30:53.492 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 344414208 bytes 2022-04-10T01:30:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 362576896 bytes 2022-04-10T01:30:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 398754816 bytes 2022-04-10T01:30:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 463517696 bytes 2022-04-10T01:30:54.066 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 509947904 bytes 2022-04-10T01:30:54.244 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 572591104 bytes 2022-04-10T01:30:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 585802752 bytes 2022-04-10T01:30:54.492 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 621644800 bytes 2022-04-10T01:30:54.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 640182272 bytes 2022-04-10T01:30:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 658647040 bytes 2022-04-10T01:30:54.903 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 713152512 bytes 2022-04-10T01:30:55.008 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 725569536 bytes 2022-04-10T01:30:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 785710080 bytes 2022-04-10T01:30:55.309 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 819082240 bytes 2022-04-10T01:30:55.445 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 851319808 bytes 2022-04-10T01:30:55.615 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 918133760 bytes 2022-04-10T01:30:55.781 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 988279808 bytes 2022-04-10T01:30:55.951 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 1042163712 bytes 2022-04-10T01:30:56.064 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1055915008 bytes 2022-04-10T01:30:56.179 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1067755520 bytes 2022-04-10T01:30:56.351 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1120650240 bytes 2022-04-10T01:30:56.464 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1135310848 bytes 2022-04-10T01:30:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1171388416 bytes 2022-04-10T01:30:56.753 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1218239488 bytes 2022-04-10T01:30:56.926 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1291728896 bytes 2022-04-10T01:30:57.079 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1330087936 bytes 2022-04-10T01:30:57.247 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1377418240 bytes 2022-04-10T01:30:57.353 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1380618240 bytes 2022-04-10T01:30:57.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1429486592 bytes 2022-04-10T01:30:57.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1472547840 bytes 2022-04-10T01:30:57.823 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1524695040 bytes 2022-04-10T01:30:57.980 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1573916672 bytes 2022-04-10T01:30:58.125 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1624043520 bytes 2022-04-10T01:30:58.287 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1660741632 bytes 2022-04-10T01:30:58.309 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:30:58.309 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:30:58.309 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:30:58.309 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:30:58.310 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:30:58.310 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_17 2022-04-10T01:30:58.310 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image_and_enable_mirror cluster2 mirror image_17 snapshot 128M 2022-04-10T01:30:58.310 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:30:58.311 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:30:58.311 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:30:58.311 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:30:58.311 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_17 2022-04-10T01:30:58.312 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:30:58.312 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:30:58.312 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:30:58.312 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:30:58.312 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:30:58.313 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:30:58.313 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snapshot ']' 2022-04-10T01:30:58.313 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:30:58.313 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_17 128M 2022-04-10T01:30:58.314 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:30:58.314 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:30:58.314 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:30:58.314 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:30:58.314 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_17 2022-04-10T01:30:58.315 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:30:58.315 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:30:58.315 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:30:58.315 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:30:58.316 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:30:58.317 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:30:58.317 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 128M ']' 2022-04-10T01:30:58.318 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=128M 2022-04-10T01:30:58.318 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:30:58.318 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_17 2022-04-10T01:30:58.347 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' image = image ']' 2022-04-10T01:30:58.348 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_17 snapshot 2022-04-10T01:30:58.348 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:30:58.348 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:30:58.349 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_17 2022-04-10T01:30:58.349 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:30:58.349 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:30:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:30:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:30:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:30:58.351 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_17 snapshot 2022-04-10T01:30:58.439 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:30:58.447 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_17 2022-04-10T01:30:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_17': 2022-04-10T01:30:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:30:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:30:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:30:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 12f17977d9e5 2022-04-10T01:30:58.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.12f17977d9e5 2022-04-10T01:30:58.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:30:58.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:30:58.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:30:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:30:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:30:58 2022 2022-04-10T01:30:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:30:58 2022 2022-04-10T01:30:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:30:58 2022 2022-04-10T01:30:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:30:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:30:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 59134803-ff09-429b-9467-2cdbc3d6d23c 2022-04-10T01:30:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:30:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:30:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_17 2022-04-10T01:30:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:30:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:30:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_17 2022-04-10T01:30:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:30:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=8 2022-04-10T01:30:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:30:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:30:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:30:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:30:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:30:58.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_17 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:30:58.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 68353024 bytes 2022-04-10T01:30:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 89788416 bytes 2022-04-10T01:30:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 119665664 bytes 2022-04-10T01:30:59.036 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 120275968 bytes 2022-04-10T01:30:59.164 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 143909888 bytes 2022-04-10T01:30:59.273 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 163386368 bytes 2022-04-10T01:30:59.423 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 197184512 bytes 2022-04-10T01:30:59.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 225526784 bytes 2022-04-10T01:30:59.648 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 226131968 bytes 2022-04-10T01:30:59.769 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 244557824 bytes 2022-04-10T01:30:59.867 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 245174272 bytes 2022-04-10T01:30:59.977 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 258451456 bytes 2022-04-10T01:31:00.077 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 263097344 bytes 2022-04-10T01:31:00.244 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 316750848 bytes 2022-04-10T01:31:00.370 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 336287744 bytes 2022-04-10T01:31:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 373218304 bytes 2022-04-10T01:31:00.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 414655488 bytes 2022-04-10T01:31:00.772 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 421233664 bytes 2022-04-10T01:31:01.008 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 529274880 bytes 2022-04-10T01:31:01.024 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.2516643196221663) > weight (0.25) 2022-04-10T01:31:01.024 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.3438093395249351) > weight (0.25) 2022-04-10T01:31:01.025 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.42182018394258614) > weight (0.25) 2022-04-10T01:31:01.025 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.8561291023694649) > weight (0.25) 2022-04-10T01:31:01.025 INFO:tasks.rbd_mirror_thrash:waiting for 118 secs before thrashing 2022-04-10T01:31:01.134 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 552936448 bytes 2022-04-10T01:31:01.350 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 636846080 bytes 2022-04-10T01:31:01.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 717545472 bytes 2022-04-10T01:31:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 745444352 bytes 2022-04-10T01:31:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 780553216 bytes 2022-04-10T01:31:01.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 802585600 bytes 2022-04-10T01:31:02.047 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 810640384 bytes 2022-04-10T01:31:02.222 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 869726208 bytes 2022-04-10T01:31:02.366 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 905396224 bytes 2022-04-10T01:31:02.501 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 931808256 bytes 2022-04-10T01:31:02.627 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 961259520 bytes 2022-04-10T01:31:02.849 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1056323584 bytes 2022-04-10T01:31:02.984 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1099133952 bytes 2022-04-10T01:31:03.131 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1133425664 bytes 2022-04-10T01:31:03.296 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1176698880 bytes 2022-04-10T01:31:03.458 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1228792832 bytes 2022-04-10T01:31:03.583 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1268883456 bytes 2022-04-10T01:31:03.723 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1284850688 bytes 2022-04-10T01:31:03.848 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1329801216 bytes 2022-04-10T01:31:04.123 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1461167104 bytes 2022-04-10T01:31:04.332 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1551141888 bytes 2022-04-10T01:31:04.461 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1573511168 bytes 2022-04-10T01:31:04.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 1644908544 bytes 2022-04-10T01:31:04.846 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 1721756672 bytes 2022-04-10T01:31:05.037 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 1798619136 bytes 2022-04-10T01:31:05.196 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 1861364736 bytes 2022-04-10T01:31:05.339 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 1880970240 bytes 2022-04-10T01:31:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 1904915456 bytes 2022-04-10T01:31:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 1950917632 bytes 2022-04-10T01:31:05.730 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 1951533056 bytes 2022-04-10T01:31:05.892 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 2028994560 bytes 2022-04-10T01:31:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 2076929024 bytes 2022-04-10T01:31:06.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 2110883840 bytes 2022-04-10T01:31:06.346 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 2151036928 bytes 2022-04-10T01:31:06.491 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:31:06.491 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:31:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:31:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:31:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:31:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_18 2022-04-10T01:31:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image_and_enable_mirror cluster2 mirror image_18 snapshot 128M 2022-04-10T01:31:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:31:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:31:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:31:06.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:31:06.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_18 2022-04-10T01:31:06.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:31:06.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:31:06.495 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:31:06.495 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:31:06.495 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:31:06.495 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:31:06.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snapshot ']' 2022-04-10T01:31:06.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:31:06.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_18 128M 2022-04-10T01:31:06.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:31:06.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:31:06.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:31:06.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:31:06.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_18 2022-04-10T01:31:06.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:31:06.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:31:06.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:31:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:31:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:31:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:31:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 128M ']' 2022-04-10T01:31:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=128M 2022-04-10T01:31:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:31:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_18 2022-04-10T01:31:06.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' image = image ']' 2022-04-10T01:31:06.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_18 snapshot 2022-04-10T01:31:06.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:31:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:31:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_18 2022-04-10T01:31:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:31:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:31:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:31:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:31:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:31:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_18 snapshot 2022-04-10T01:31:06.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:31:06.660 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_18 2022-04-10T01:31:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_18': 2022-04-10T01:31:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:31:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:31:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:31:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 12fb47ebc28f 2022-04-10T01:31:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.12fb47ebc28f 2022-04-10T01:31:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:31:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:31:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:31:06.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:31:06.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:31:06 2022 2022-04-10T01:31:06.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:31:06 2022 2022-04-10T01:31:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:31:06 2022 2022-04-10T01:31:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:31:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:31:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 0481cc7e-2bc7-4c1c-903d-89babd0e91d4 2022-04-10T01:31:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:31:06.699 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:31:06.699 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_18 2022-04-10T01:31:06.699 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:31:06.700 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:31:06.700 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_18 2022-04-10T01:31:06.700 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:31:06.700 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=10 2022-04-10T01:31:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:31:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:31:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:31:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:31:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:31:06.702 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_18 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:31:06.831 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 611328 bytes 2022-04-10T01:31:06.946 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 27109376 bytes 2022-04-10T01:31:07.054 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 40428544 bytes 2022-04-10T01:31:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 128964608 bytes 2022-04-10T01:31:07.376 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 153171968 bytes 2022-04-10T01:31:07.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 220596224 bytes 2022-04-10T01:31:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 241814528 bytes 2022-04-10T01:31:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 277364736 bytes 2022-04-10T01:31:07.956 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 315992064 bytes 2022-04-10T01:31:08.108 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 362387456 bytes 2022-04-10T01:31:08.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 405180416 bytes 2022-04-10T01:31:08.452 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 481138688 bytes 2022-04-10T01:31:08.608 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 536246272 bytes 2022-04-10T01:31:08.759 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 561836032 bytes 2022-04-10T01:31:08.908 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 602737664 bytes 2022-04-10T01:31:09.063 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 668811264 bytes 2022-04-10T01:31:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 738192384 bytes 2022-04-10T01:31:09.363 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 767197184 bytes 2022-04-10T01:31:09.500 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 794619904 bytes 2022-04-10T01:31:09.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 866826240 bytes 2022-04-10T01:31:09.845 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 908500992 bytes 2022-04-10T01:31:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 909406208 bytes 2022-04-10T01:31:10.064 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 923481088 bytes 2022-04-10T01:31:10.196 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 954822656 bytes 2022-04-10T01:31:10.374 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 956442624 bytes 2022-04-10T01:31:10.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1014285312 bytes 2022-04-10T01:31:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1036192768 bytes 2022-04-10T01:31:10.788 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1045953536 bytes 2022-04-10T01:31:10.911 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1065339904 bytes 2022-04-10T01:31:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1132961792 bytes 2022-04-10T01:31:11.284 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1189139456 bytes 2022-04-10T01:31:11.393 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1196509184 bytes 2022-04-10T01:31:11.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1222791168 bytes 2022-04-10T01:31:11.725 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1291819008 bytes 2022-04-10T01:31:11.868 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1329349632 bytes 2022-04-10T01:31:11.977 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1338505216 bytes 2022-04-10T01:31:12.129 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1382025216 bytes 2022-04-10T01:31:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1441718272 bytes 2022-04-10T01:31:12.426 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1463784448 bytes 2022-04-10T01:31:12.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1487010816 bytes 2022-04-10T01:31:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1487610880 bytes 2022-04-10T01:31:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 1494835200 bytes 2022-04-10T01:31:12.916 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 1533499392 bytes 2022-04-10T01:31:13.063 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 1572675584 bytes 2022-04-10T01:31:13.199 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 1610085376 bytes 2022-04-10T01:31:13.312 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 1618714624 bytes 2022-04-10T01:31:13.493 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 1680690176 bytes 2022-04-10T01:31:13.649 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 1720495104 bytes 2022-04-10T01:31:13.889 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 1855607808 bytes 2022-04-10T01:31:14.073 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 1902156800 bytes 2022-04-10T01:31:14.247 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 1959437312 bytes 2022-04-10T01:31:14.416 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 2012959744 bytes 2022-04-10T01:31:14.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 2060646400 bytes 2022-04-10T01:31:14.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5400 IOs, 2086583296 bytes 2022-04-10T01:31:14.910 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5500 IOs, 2168818688 bytes 2022-04-10T01:31:15.054 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5600 IOs, 2204665856 bytes 2022-04-10T01:31:15.218 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5700 IOs, 2265984000 bytes 2022-04-10T01:31:15.334 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5800 IOs, 2270557184 bytes 2022-04-10T01:31:15.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5900 IOs, 2342463488 bytes 2022-04-10T01:31:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6000 IOs, 2373412864 bytes 2022-04-10T01:31:15.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6100 IOs, 2466191360 bytes 2022-04-10T01:31:16.036 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6200 IOs, 2520271872 bytes 2022-04-10T01:31:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6300 IOs, 2581003264 bytes 2022-04-10T01:31:16.350 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6400 IOs, 2617287680 bytes 2022-04-10T01:31:16.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6500 IOs, 2658488320 bytes 2022-04-10T01:31:16.630 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6600 IOs, 2677253120 bytes 2022-04-10T01:31:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:31:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:31:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:31:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:31:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:31:16.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_19 2022-04-10T01:31:16.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image_and_enable_mirror cluster2 mirror image_19 snapshot 128M 2022-04-10T01:31:16.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:31:16.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:31:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:31:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:31:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_19 2022-04-10T01:31:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:31:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:31:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:31:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:31:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:31:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:31:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snapshot ']' 2022-04-10T01:31:16.713 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:31:16.713 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_19 128M 2022-04-10T01:31:16.713 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:31:16.713 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:31:16.714 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:31:16.714 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:31:16.714 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_19 2022-04-10T01:31:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:31:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:31:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:31:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:31:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:31:16.716 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:31:16.717 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 128M ']' 2022-04-10T01:31:16.717 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=128M 2022-04-10T01:31:16.717 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:31:16.717 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_19 2022-04-10T01:31:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' image = image ']' 2022-04-10T01:31:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_19 snapshot 2022-04-10T01:31:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:31:16.758 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:31:16.758 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_19 2022-04-10T01:31:16.758 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:31:16.758 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:31:16.759 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:31:16.759 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:31:16.759 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:31:16.759 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_19 snapshot 2022-04-10T01:31:16.876 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:31:16.884 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_19 2022-04-10T01:31:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_19': 2022-04-10T01:31:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:31:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:31:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:31:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 130491ad8a7f 2022-04-10T01:31:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.130491ad8a7f 2022-04-10T01:31:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:31:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:31:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:31:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:31:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:31:16 2022 2022-04-10T01:31:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:31:16 2022 2022-04-10T01:31:16.921 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:31:16 2022 2022-04-10T01:31:16.921 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:31:16.921 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:31:16.921 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 45a317c4-845d-40d4-b7de-aa8c3ca7ce90 2022-04-10T01:31:16.922 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:31:16.924 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:31:16.924 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_19 2022-04-10T01:31:16.924 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:31:16.924 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:31:16.925 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_19 2022-04-10T01:31:16.925 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:31:16.925 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=8 2022-04-10T01:31:16.925 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:31:16.925 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:31:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:31:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:31:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:31:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_19 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:31:17.057 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 605184 bytes 2022-04-10T01:31:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 48430080 bytes 2022-04-10T01:31:17.324 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 78270464 bytes 2022-04-10T01:31:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 119234560 bytes 2022-04-10T01:31:17.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 132039680 bytes 2022-04-10T01:31:17.747 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 189707264 bytes 2022-04-10T01:31:17.897 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 250595328 bytes 2022-04-10T01:31:18.067 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 303083520 bytes 2022-04-10T01:31:18.220 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 353695744 bytes 2022-04-10T01:31:18.348 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 382372864 bytes 2022-04-10T01:31:18.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 427051008 bytes 2022-04-10T01:31:18.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 484992000 bytes 2022-04-10T01:31:18.783 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 508121088 bytes 2022-04-10T01:31:18.883 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 508720128 bytes 2022-04-10T01:31:19.012 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 541193216 bytes 2022-04-10T01:31:19.153 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 577018880 bytes 2022-04-10T01:31:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 632714240 bytes 2022-04-10T01:31:19.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 680574976 bytes 2022-04-10T01:31:19.619 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 727438336 bytes 2022-04-10T01:31:19.831 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 812998656 bytes 2022-04-10T01:31:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 883171328 bytes 2022-04-10T01:31:20.159 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 928660480 bytes 2022-04-10T01:31:20.275 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 942017536 bytes 2022-04-10T01:31:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 949708800 bytes 2022-04-10T01:31:20.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 1004589056 bytes 2022-04-10T01:31:20.723 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1070116864 bytes 2022-04-10T01:31:20.869 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1132960768 bytes 2022-04-10T01:31:21.033 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1167310848 bytes 2022-04-10T01:31:21.141 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1177369600 bytes 2022-04-10T01:31:21.262 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1200321536 bytes 2022-04-10T01:31:21.426 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1253430272 bytes 2022-04-10T01:31:21.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1287396352 bytes 2022-04-10T01:31:21.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1316808704 bytes 2022-04-10T01:31:21.827 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3672742381142782) > weight (0.25) 2022-04-10T01:31:21.828 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.38082548599863164) > weight (0.25) 2022-04-10T01:31:21.828 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.9124121493247077) > weight (0.25) 2022-04-10T01:31:21.829 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.6270596931945828) > weight (0.25) 2022-04-10T01:31:21.829 INFO:tasks.rbd_mirror_thrash:waiting for 115 secs before thrashing 2022-04-10T01:31:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1380780032 bytes 2022-04-10T01:31:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1390366720 bytes 2022-04-10T01:31:22.104 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1411904512 bytes 2022-04-10T01:31:22.230 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1435204608 bytes 2022-04-10T01:31:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1459597312 bytes 2022-04-10T01:31:22.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1477707776 bytes 2022-04-10T01:31:22.583 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1501271040 bytes 2022-04-10T01:31:22.784 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1572765696 bytes 2022-04-10T01:31:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 1584203776 bytes 2022-04-10T01:31:23.089 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 1666457600 bytes 2022-04-10T01:31:23.200 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 1680092160 bytes 2022-04-10T01:31:23.325 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 1709158400 bytes 2022-04-10T01:31:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 1738907648 bytes 2022-04-10T01:31:23.600 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 1765963776 bytes 2022-04-10T01:31:23.778 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 1831762944 bytes 2022-04-10T01:31:23.911 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 1864701952 bytes 2022-04-10T01:31:24.029 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 1880005632 bytes 2022-04-10T01:31:24.193 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 1927432192 bytes 2022-04-10T01:31:24.315 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 1948556288 bytes 2022-04-10T01:31:24.426 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 1962846208 bytes 2022-04-10T01:31:24.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5400 IOs, 2004892672 bytes 2022-04-10T01:31:24.715 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5500 IOs, 2035802112 bytes 2022-04-10T01:31:24.893 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5600 IOs, 2105861120 bytes 2022-04-10T01:31:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:31:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:31:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:31:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:31:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:31:24.932 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_20 2022-04-10T01:31:24.932 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image_and_enable_mirror cluster2 mirror image_20 snapshot 128M 2022-04-10T01:31:24.932 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:31:24.932 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:31:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:31:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:31:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_20 2022-04-10T01:31:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:31:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:31:24.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:31:24.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:31:24.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:31:24.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:31:24.935 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snapshot ']' 2022-04-10T01:31:24.935 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:31:24.935 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_20 128M 2022-04-10T01:31:24.935 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:31:24.936 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:31:24.936 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:31:24.936 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:31:24.937 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_20 2022-04-10T01:31:24.937 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:31:24.937 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:31:24.937 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:31:24.937 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:31:24.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:31:24.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:31:24.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 128M ']' 2022-04-10T01:31:24.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=128M 2022-04-10T01:31:24.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:31:24.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_20 2022-04-10T01:31:24.970 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' image = image ']' 2022-04-10T01:31:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_20 snapshot 2022-04-10T01:31:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:31:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:31:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_20 2022-04-10T01:31:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:31:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:31:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:31:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:31:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:31:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_20 snapshot 2022-04-10T01:31:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:31:25.354 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_20 2022-04-10T01:31:25.387 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_20': 2022-04-10T01:31:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:31:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:31:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:31:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 130e5e6f5e 2022-04-10T01:31:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.130e5e6f5e 2022-04-10T01:31:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:31:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:31:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:31:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:31:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:31:24 2022 2022-04-10T01:31:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:31:24 2022 2022-04-10T01:31:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:31:24 2022 2022-04-10T01:31:25.391 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:31:25.391 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:31:25.391 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 828304b8-d69a-488c-b09a-e6569fe239bd 2022-04-10T01:31:25.391 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:31:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:31:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_20 2022-04-10T01:31:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:31:25.394 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:31:25.394 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_20 2022-04-10T01:31:25.394 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:31:25.394 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=8 2022-04-10T01:31:25.395 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:31:25.395 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:31:25.395 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:31:25.395 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:31:25.396 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:31:25.396 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_20 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:31:25.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 51876864 bytes 2022-04-10T01:31:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 71184384 bytes 2022-04-10T01:31:25.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 73616384 bytes 2022-04-10T01:31:25.953 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 114668544 bytes 2022-04-10T01:31:26.101 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 177616896 bytes 2022-04-10T01:31:26.273 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 224592896 bytes 2022-04-10T01:31:26.432 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 270483456 bytes 2022-04-10T01:31:26.607 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 333273088 bytes 2022-04-10T01:31:26.791 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 403658752 bytes 2022-04-10T01:31:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 420013056 bytes 2022-04-10T01:31:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 449583104 bytes 2022-04-10T01:31:27.214 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 507839488 bytes 2022-04-10T01:31:27.314 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 508423168 bytes 2022-04-10T01:31:27.462 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 566680576 bytes 2022-04-10T01:31:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 613276672 bytes 2022-04-10T01:31:27.774 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 643089408 bytes 2022-04-10T01:31:27.877 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 643681280 bytes 2022-04-10T01:31:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 742414336 bytes 2022-04-10T01:31:28.222 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 761966592 bytes 2022-04-10T01:31:28.372 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 807603200 bytes 2022-04-10T01:31:28.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 818277376 bytes 2022-04-10T01:31:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 818889728 bytes 2022-04-10T01:31:28.730 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 852998144 bytes 2022-04-10T01:31:28.873 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 891920384 bytes 2022-04-10T01:31:29.054 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 959926272 bytes 2022-04-10T01:31:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1028776960 bytes 2022-04-10T01:31:29.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1069648896 bytes 2022-04-10T01:31:29.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1092984832 bytes 2022-04-10T01:31:29.627 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1105872896 bytes 2022-04-10T01:31:29.738 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1114295296 bytes 2022-04-10T01:31:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1154680832 bytes 2022-04-10T01:31:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1183808512 bytes 2022-04-10T01:31:30.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1259607040 bytes 2022-04-10T01:31:30.348 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1281485824 bytes 2022-04-10T01:31:30.501 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1323540480 bytes 2022-04-10T01:31:30.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1382790144 bytes 2022-04-10T01:31:30.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1472376832 bytes 2022-04-10T01:31:31.028 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1502057472 bytes 2022-04-10T01:31:31.242 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1589599232 bytes 2022-04-10T01:31:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1618415616 bytes 2022-04-10T01:31:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1632875520 bytes 2022-04-10T01:31:31.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 1677280256 bytes 2022-04-10T01:31:31.762 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 1704375296 bytes 2022-04-10T01:31:31.937 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 1755029504 bytes 2022-04-10T01:31:32.048 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 1765986304 bytes 2022-04-10T01:31:32.151 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 1766693888 bytes 2022-04-10T01:31:32.336 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 1836053504 bytes 2022-04-10T01:31:32.483 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 1873449984 bytes 2022-04-10T01:31:32.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 1919925248 bytes 2022-04-10T01:31:32.810 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 1967239168 bytes 2022-04-10T01:31:32.954 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 2013145088 bytes 2022-04-10T01:31:33.096 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 2046414848 bytes 2022-04-10T01:31:33.289 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 2121256960 bytes 2022-04-10T01:31:33.401 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:31:33.402 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:31:33.402 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:31:33.402 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:31:33.403 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:31:33.403 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_21 2022-04-10T01:31:33.403 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image_and_enable_mirror cluster2 mirror image_21 snapshot 128M 2022-04-10T01:31:33.404 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:31:33.404 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:31:33.404 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:31:33.404 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:31:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_21 2022-04-10T01:31:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:31:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:31:33.406 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:31:33.406 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:31:33.406 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:31:33.406 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:31:33.406 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snapshot ']' 2022-04-10T01:31:33.407 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:31:33.407 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_21 128M 2022-04-10T01:31:33.407 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:31:33.407 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:31:33.407 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:31:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:31:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_21 2022-04-10T01:31:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:31:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:31:33.409 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:31:33.409 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:31:33.409 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:31:33.409 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:31:33.409 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 128M ']' 2022-04-10T01:31:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=128M 2022-04-10T01:31:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:31:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_21 2022-04-10T01:31:33.445 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' image = image ']' 2022-04-10T01:31:33.445 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_21 snapshot 2022-04-10T01:31:33.445 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:31:33.446 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:31:33.446 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_21 2022-04-10T01:31:33.446 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:31:33.446 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:31:33.446 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:31:33.447 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:31:33.447 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:31:33.447 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_21 snapshot 2022-04-10T01:31:34.392 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:31:34.400 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_21 2022-04-10T01:31:34.437 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_21': 2022-04-10T01:31:34.437 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:31:34.437 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:31:34.438 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:31:34.438 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 1317f44bc6cf 2022-04-10T01:31:34.438 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.1317f44bc6cf 2022-04-10T01:31:34.438 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:31:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:31:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:31:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:31:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:31:33 2022 2022-04-10T01:31:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:31:33 2022 2022-04-10T01:31:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:31:33 2022 2022-04-10T01:31:34.441 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:31:34.441 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:31:34.441 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 65c94954-3314-4812-9697-85faa0cd8459 2022-04-10T01:31:34.442 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:31:34.443 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:31:34.444 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_21 2022-04-10T01:31:34.444 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:31:34.444 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:31:34.444 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_21 2022-04-10T01:31:34.445 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:31:34.445 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=5 2022-04-10T01:31:34.445 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:31:34.445 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:31:34.445 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:31:34.446 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:31:34.446 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:31:34.446 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_21 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:31:34.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 63042560 bytes 2022-04-10T01:31:34.863 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 159991808 bytes 2022-04-10T01:31:35.048 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 205984768 bytes 2022-04-10T01:31:35.212 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 265550848 bytes 2022-04-10T01:31:35.333 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 273745920 bytes 2022-04-10T01:31:35.490 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 317767680 bytes 2022-04-10T01:31:35.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 376536064 bytes 2022-04-10T01:31:35.773 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 385620992 bytes 2022-04-10T01:31:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 406005760 bytes 2022-04-10T01:31:36.108 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 498639872 bytes 2022-04-10T01:31:36.265 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 550804480 bytes 2022-04-10T01:31:36.429 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 593958912 bytes 2022-04-10T01:31:36.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 627409920 bytes 2022-04-10T01:31:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 707198976 bytes 2022-04-10T01:31:36.932 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 762525696 bytes 2022-04-10T01:31:37.051 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 794703872 bytes 2022-04-10T01:31:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 839442432 bytes 2022-04-10T01:31:37.368 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 894147584 bytes 2022-04-10T01:31:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 907105280 bytes 2022-04-10T01:31:37.619 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 914600960 bytes 2022-04-10T01:31:37.736 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 938446848 bytes 2022-04-10T01:31:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 962393088 bytes 2022-04-10T01:31:38.061 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 1012918272 bytes 2022-04-10T01:31:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 1060960256 bytes 2022-04-10T01:31:38.363 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 1102759936 bytes 2022-04-10T01:31:38.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1140198400 bytes 2022-04-10T01:31:38.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1197601792 bytes 2022-04-10T01:31:38.839 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1274228736 bytes 2022-04-10T01:31:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1338248192 bytes 2022-04-10T01:31:39.233 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1393153024 bytes 2022-04-10T01:31:39.389 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1434766336 bytes 2022-04-10T01:31:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:31:39.455 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:31:39.455 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:31:39.455 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:31:39.456 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:31:39.456 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_22 2022-04-10T01:31:39.456 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image_and_enable_mirror cluster2 mirror image_22 snapshot 128M 2022-04-10T01:31:39.456 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:31:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:31:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:31:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:31:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_22 2022-04-10T01:31:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:31:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:31:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:31:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:31:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:31:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:31:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snapshot ']' 2022-04-10T01:31:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:31:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_22 128M 2022-04-10T01:31:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:31:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:31:39.461 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:31:39.461 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:31:39.461 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_22 2022-04-10T01:31:39.461 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:31:39.461 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:31:39.462 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:31:39.462 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:31:39.462 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:31:39.462 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:31:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 128M ']' 2022-04-10T01:31:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=128M 2022-04-10T01:31:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:31:39.464 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_22 2022-04-10T01:31:39.495 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' image = image ']' 2022-04-10T01:31:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_22 snapshot 2022-04-10T01:31:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:31:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:31:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_22 2022-04-10T01:31:39.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:31:39.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:31:39.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:31:39.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:31:39.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:31:39.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_22 snapshot 2022-04-10T01:31:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:31:40.360 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_22 2022-04-10T01:31:40.397 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_22': 2022-04-10T01:31:40.397 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:31:40.397 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:31:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:31:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 13215ee9af0d 2022-04-10T01:31:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.13215ee9af0d 2022-04-10T01:31:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:31:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:31:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:31:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:31:40.400 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:31:39 2022 2022-04-10T01:31:40.400 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:31:39 2022 2022-04-10T01:31:40.401 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:31:39 2022 2022-04-10T01:31:40.401 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:31:40.401 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:31:40.401 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: be4c9569-5fe3-47a0-a0a1-84d380d56af1 2022-04-10T01:31:40.402 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:31:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:31:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_22 2022-04-10T01:31:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:31:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:31:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_22 2022-04-10T01:31:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:31:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=13 2022-04-10T01:31:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:31:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:31:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:31:40.406 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:31:40.406 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:31:40.406 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_22 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:31:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 46032896 bytes 2022-04-10T01:31:40.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 121700352 bytes 2022-04-10T01:31:40.911 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 122295296 bytes 2022-04-10T01:31:41.184 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 212144128 bytes 2022-04-10T01:31:41.435 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 269857792 bytes 2022-04-10T01:31:41.590 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 316685312 bytes 2022-04-10T01:31:41.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 318381056 bytes 2022-04-10T01:31:41.871 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 382811136 bytes 2022-04-10T01:31:41.999 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 407205888 bytes 2022-04-10T01:31:42.142 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 439088128 bytes 2022-04-10T01:31:42.320 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 506755072 bytes 2022-04-10T01:31:42.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 561967104 bytes 2022-04-10T01:31:42.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 623589376 bytes 2022-04-10T01:31:42.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 709486592 bytes 2022-04-10T01:31:43.078 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 757220352 bytes 2022-04-10T01:31:43.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 812231680 bytes 2022-04-10T01:31:43.372 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 827067392 bytes 2022-04-10T01:31:43.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 897507328 bytes 2022-04-10T01:31:43.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 914527232 bytes 2022-04-10T01:31:43.819 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 951625728 bytes 2022-04-10T01:31:43.989 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 1022288896 bytes 2022-04-10T01:31:44.174 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 1084267520 bytes 2022-04-10T01:31:44.365 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 1172328448 bytes 2022-04-10T01:31:44.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 1225474048 bytes 2022-04-10T01:31:44.706 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 1268899840 bytes 2022-04-10T01:31:44.864 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1328375808 bytes 2022-04-10T01:31:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1363495936 bytes 2022-04-10T01:31:45.175 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1409296384 bytes 2022-04-10T01:31:45.308 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1456468992 bytes 2022-04-10T01:31:45.474 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1494376448 bytes 2022-04-10T01:31:45.627 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1525510144 bytes 2022-04-10T01:31:45.823 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1607325696 bytes 2022-04-10T01:31:45.987 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1653236736 bytes 2022-04-10T01:31:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1709047808 bytes 2022-04-10T01:31:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1733371904 bytes 2022-04-10T01:31:46.398 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1745642496 bytes 2022-04-10T01:31:46.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1767801856 bytes 2022-04-10T01:31:46.649 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1785578496 bytes 2022-04-10T01:31:46.761 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1798088704 bytes 2022-04-10T01:31:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1841488896 bytes 2022-04-10T01:31:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1859721216 bytes 2022-04-10T01:31:47.196 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 1907400704 bytes 2022-04-10T01:31:47.343 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 1951336448 bytes 2022-04-10T01:31:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 1951954944 bytes 2022-04-10T01:31:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 2017176576 bytes 2022-04-10T01:31:47.732 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 2017786880 bytes 2022-04-10T01:31:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 2093701120 bytes 2022-04-10T01:31:48.048 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 2108072960 bytes 2022-04-10T01:31:48.167 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 2120476672 bytes 2022-04-10T01:31:48.302 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 2148943872 bytes 2022-04-10T01:31:48.436 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 2174398464 bytes 2022-04-10T01:31:48.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 2241945600 bytes 2022-04-10T01:31:48.779 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 2289581056 bytes 2022-04-10T01:31:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5400 IOs, 2290214912 bytes 2022-04-10T01:31:49.044 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5500 IOs, 2336971776 bytes 2022-04-10T01:31:49.182 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5600 IOs, 2374917120 bytes 2022-04-10T01:31:49.335 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5700 IOs, 2399810560 bytes 2022-04-10T01:31:49.446 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5800 IOs, 2400421888 bytes 2022-04-10T01:31:49.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5900 IOs, 2427318272 bytes 2022-04-10T01:31:49.713 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6000 IOs, 2470103040 bytes 2022-04-10T01:31:49.943 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6100 IOs, 2569245696 bytes 2022-04-10T01:31:50.153 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6200 IOs, 2645341184 bytes 2022-04-10T01:31:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6300 IOs, 2645942272 bytes 2022-04-10T01:31:50.384 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6400 IOs, 2668528640 bytes 2022-04-10T01:31:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6500 IOs, 2713532416 bytes 2022-04-10T01:31:50.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6600 IOs, 2738186240 bytes 2022-04-10T01:31:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6700 IOs, 2760462336 bytes 2022-04-10T01:31:51.043 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6800 IOs, 2876356608 bytes 2022-04-10T01:31:51.209 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6900 IOs, 2918087680 bytes 2022-04-10T01:31:51.358 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7000 IOs, 2970054656 bytes 2022-04-10T01:31:51.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7100 IOs, 3114038272 bytes 2022-04-10T01:31:51.759 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7200 IOs, 3123019776 bytes 2022-04-10T01:31:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7300 IOs, 3152319488 bytes 2022-04-10T01:31:52.003 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7400 IOs, 3152945152 bytes 2022-04-10T01:31:52.141 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7500 IOs, 3178392576 bytes 2022-04-10T01:31:52.282 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7600 IOs, 3205122048 bytes 2022-04-10T01:31:52.484 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7700 IOs, 3291832320 bytes 2022-04-10T01:31:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7800 IOs, 3300204544 bytes 2022-04-10T01:31:52.736 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7900 IOs, 3347818496 bytes 2022-04-10T01:31:52.927 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8000 IOs, 3396570112 bytes 2022-04-10T01:31:53.124 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8100 IOs, 3471734784 bytes 2022-04-10T01:31:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8200 IOs, 3495346176 bytes 2022-04-10T01:31:53.411 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:31:53.412 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:31:53.412 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:31:53.412 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:31:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:31:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_23 2022-04-10T01:31:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image_and_enable_mirror cluster2 mirror image_23 snapshot 128M 2022-04-10T01:31:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:31:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:31:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:31:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:31:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_23 2022-04-10T01:31:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:31:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:31:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:31:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:31:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:31:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:31:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snapshot ']' 2022-04-10T01:31:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:31:53.418 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_23 128M 2022-04-10T01:31:53.418 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:31:53.418 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:31:53.419 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:31:53.419 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:31:53.419 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_23 2022-04-10T01:31:53.419 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:31:53.420 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:31:53.420 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:31:53.420 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:31:53.420 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:31:53.421 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:31:53.421 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 128M ']' 2022-04-10T01:31:53.421 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=128M 2022-04-10T01:31:53.421 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:31:53.422 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_23 2022-04-10T01:31:53.455 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' image = image ']' 2022-04-10T01:31:53.456 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_23 snapshot 2022-04-10T01:31:53.456 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:31:53.456 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:31:53.457 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_23 2022-04-10T01:31:53.457 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:31:53.458 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:31:53.458 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:31:53.458 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:31:53.459 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:31:53.459 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_23 snapshot 2022-04-10T01:31:54.392 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:31:54.400 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_23 2022-04-10T01:31:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_23': 2022-04-10T01:31:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:31:54.438 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:31:54.438 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:31:54.438 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 132b1c8a36a2 2022-04-10T01:31:54.438 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.132b1c8a36a2 2022-04-10T01:31:54.439 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:31:54.439 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:31:54.439 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:31:54.439 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:31:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:31:53 2022 2022-04-10T01:31:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:31:53 2022 2022-04-10T01:31:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:31:53 2022 2022-04-10T01:31:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:31:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:31:54.441 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 656c53f5-50a2-44ea-8dbc-608ddbbbe7f5 2022-04-10T01:31:54.441 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:31:54.443 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:31:54.443 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_23 2022-04-10T01:31:54.443 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:31:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:31:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_23 2022-04-10T01:31:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:31:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=9 2022-04-10T01:31:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:31:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:31:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:31:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:31:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:31:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_23 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:31:54.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 100547584 bytes 2022-04-10T01:31:54.847 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 142942208 bytes 2022-04-10T01:31:55.020 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 216483840 bytes 2022-04-10T01:31:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 276861952 bytes 2022-04-10T01:31:55.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 319214592 bytes 2022-04-10T01:31:55.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 356548608 bytes 2022-04-10T01:31:55.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 396456960 bytes 2022-04-10T01:31:55.795 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 397047808 bytes 2022-04-10T01:31:56.040 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 517500928 bytes 2022-04-10T01:31:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 564257792 bytes 2022-04-10T01:31:56.375 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 613959680 bytes 2022-04-10T01:31:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 706512896 bytes 2022-04-10T01:31:56.756 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 765984768 bytes 2022-04-10T01:31:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 822525952 bytes 2022-04-10T01:31:57.085 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 854578176 bytes 2022-04-10T01:31:57.201 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 861719552 bytes 2022-04-10T01:31:57.338 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 885121024 bytes 2022-04-10T01:31:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 885739520 bytes 2022-04-10T01:31:57.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 907896832 bytes 2022-04-10T01:31:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 943568896 bytes 2022-04-10T01:31:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 989009920 bytes 2022-04-10T01:31:58.064 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 1047852032 bytes 2022-04-10T01:31:58.199 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 1069144064 bytes 2022-04-10T01:31:58.391 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 1133958144 bytes 2022-04-10T01:31:58.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 1186447360 bytes 2022-04-10T01:31:58.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1215270912 bytes 2022-04-10T01:31:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1215881216 bytes 2022-04-10T01:31:58.968 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1267931136 bytes 2022-04-10T01:31:59.108 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1295669248 bytes 2022-04-10T01:31:59.263 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1330656256 bytes 2022-04-10T01:31:59.412 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1365816320 bytes 2022-04-10T01:31:59.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1417876480 bytes 2022-04-10T01:31:59.760 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1483066368 bytes 2022-04-10T01:31:59.856 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1483693056 bytes 2022-04-10T01:32:00.057 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1562214400 bytes 2022-04-10T01:32:00.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1595841536 bytes 2022-04-10T01:32:00.322 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1598795776 bytes 2022-04-10T01:32:00.501 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1685915648 bytes 2022-04-10T01:32:00.681 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1723585536 bytes 2022-04-10T01:32:00.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1745671168 bytes 2022-04-10T01:32:00.965 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1774607360 bytes 2022-04-10T01:32:01.163 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 1849212928 bytes 2022-04-10T01:32:01.329 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 1893764096 bytes 2022-04-10T01:32:01.486 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 1950783488 bytes 2022-04-10T01:32:01.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 1970906112 bytes 2022-04-10T01:32:01.785 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 2007841792 bytes 2022-04-10T01:32:01.973 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 2080564224 bytes 2022-04-10T01:32:02.193 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 2163850240 bytes 2022-04-10T01:32:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 2202533888 bytes 2022-04-10T01:32:02.489 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 2229764096 bytes 2022-04-10T01:32:02.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 2285350912 bytes 2022-04-10T01:32:02.843 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 2340244480 bytes 2022-04-10T01:32:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 2340856832 bytes 2022-04-10T01:32:03.148 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5400 IOs, 2435868672 bytes 2022-04-10T01:32:03.347 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5500 IOs, 2498401280 bytes 2022-04-10T01:32:03.452 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:32:03.452 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:32:03.453 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:32:03.453 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:32:03.453 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:32:03.453 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_24 2022-04-10T01:32:03.453 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image_and_enable_mirror cluster2 mirror image_24 snapshot 128M 2022-04-10T01:32:03.454 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:32:03.454 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:32:03.454 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:32:03.454 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:32:03.455 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_24 2022-04-10T01:32:03.455 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:32:03.455 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:32:03.455 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:32:03.455 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:32:03.456 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:32:03.456 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:32:03.456 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snapshot ']' 2022-04-10T01:32:03.456 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:32:03.457 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_24 128M 2022-04-10T01:32:03.457 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:32:03.457 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:32:03.457 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:32:03.457 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:32:03.458 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_24 2022-04-10T01:32:03.458 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:32:03.458 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:32:03.458 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:32:03.458 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:32:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:32:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:32:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 128M ']' 2022-04-10T01:32:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=128M 2022-04-10T01:32:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:32:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_24 2022-04-10T01:32:03.493 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' image = image ']' 2022-04-10T01:32:03.493 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_24 snapshot 2022-04-10T01:32:03.493 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:32:03.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:32:03.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_24 2022-04-10T01:32:03.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:32:03.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:32:03.495 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:32:03.495 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:32:03.495 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:32:03.495 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_24 snapshot 2022-04-10T01:32:04.395 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:32:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_24 2022-04-10T01:32:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_24': 2022-04-10T01:32:04.441 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:32:04.441 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:32:04.441 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:32:04.441 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 1334d6b4a25d 2022-04-10T01:32:04.442 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.1334d6b4a25d 2022-04-10T01:32:04.442 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:32:04.442 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:32:04.442 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:32:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:32:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:32:03 2022 2022-04-10T01:32:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:32:03 2022 2022-04-10T01:32:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:32:03 2022 2022-04-10T01:32:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:32:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:32:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: a52ba6c5-cf55-4fb1-99eb-fa7caff7c3bd 2022-04-10T01:32:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:32:04.446 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:32:04.446 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_24 2022-04-10T01:32:04.447 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:32:04.447 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:32:04.447 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_24 2022-04-10T01:32:04.447 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:32:04.447 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=12 2022-04-10T01:32:04.448 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:32:04.448 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:32:04.448 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:32:04.448 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:32:04.449 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:32:04.449 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_24 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:32:04.603 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 13554688 bytes 2022-04-10T01:32:04.727 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 34458624 bytes 2022-04-10T01:32:04.862 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 68244480 bytes 2022-04-10T01:32:04.982 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 87511040 bytes 2022-04-10T01:32:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 139503616 bytes 2022-04-10T01:32:05.322 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 195972096 bytes 2022-04-10T01:32:05.455 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 235478016 bytes 2022-04-10T01:32:05.587 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 237166592 bytes 2022-04-10T01:32:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 312177664 bytes 2022-04-10T01:32:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 368899072 bytes 2022-04-10T01:32:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 415681536 bytes 2022-04-10T01:32:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 463022080 bytes 2022-04-10T01:32:06.390 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 491675648 bytes 2022-04-10T01:32:06.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 522477568 bytes 2022-04-10T01:32:06.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 542088192 bytes 2022-04-10T01:32:06.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 554814464 bytes 2022-04-10T01:32:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 606609408 bytes 2022-04-10T01:32:07.117 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 644238336 bytes 2022-04-10T01:32:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 681994240 bytes 2022-04-10T01:32:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 707648512 bytes 2022-04-10T01:32:07.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 709927936 bytes 2022-04-10T01:32:07.698 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 774279168 bytes 2022-04-10T01:32:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 850694144 bytes 2022-04-10T01:32:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 975803392 bytes 2022-04-10T01:32:08.332 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 1049845760 bytes 2022-04-10T01:32:08.493 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1095149568 bytes 2022-04-10T01:32:08.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1167138816 bytes 2022-04-10T01:32:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1167750144 bytes 2022-04-10T01:32:08.936 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1209964544 bytes 2022-04-10T01:32:09.138 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1290868736 bytes 2022-04-10T01:32:09.326 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1336840192 bytes 2022-04-10T01:32:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1400782848 bytes 2022-04-10T01:32:09.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1470994432 bytes 2022-04-10T01:32:09.829 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1497897984 bytes 2022-04-10T01:32:09.988 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1544816640 bytes 2022-04-10T01:32:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1623331840 bytes 2022-04-10T01:32:10.421 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1710479360 bytes 2022-04-10T01:32:10.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1775468544 bytes 2022-04-10T01:32:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1785204736 bytes 2022-04-10T01:32:10.909 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1785796608 bytes 2022-04-10T01:32:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1825111040 bytes 2022-04-10T01:32:11.184 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 1840691200 bytes 2022-04-10T01:32:11.315 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 1863121920 bytes 2022-04-10T01:32:11.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 1928797184 bytes 2022-04-10T01:32:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 1938805760 bytes 2022-04-10T01:32:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 1955097600 bytes 2022-04-10T01:32:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 1970454528 bytes 2022-04-10T01:32:12.062 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 2011233280 bytes 2022-04-10T01:32:12.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 2030834688 bytes 2022-04-10T01:32:12.362 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 2082855936 bytes 2022-04-10T01:32:12.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 2180810752 bytes 2022-04-10T01:32:12.754 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 2230269952 bytes 2022-04-10T01:32:12.935 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 2287712256 bytes 2022-04-10T01:32:13.071 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5400 IOs, 2311428096 bytes 2022-04-10T01:32:13.187 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5500 IOs, 2322851840 bytes 2022-04-10T01:32:13.372 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5600 IOs, 2386828288 bytes 2022-04-10T01:32:13.543 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5700 IOs, 2430582784 bytes 2022-04-10T01:32:13.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5800 IOs, 2450297856 bytes 2022-04-10T01:32:13.837 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5900 IOs, 2502171648 bytes 2022-04-10T01:32:14.080 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6000 IOs, 2603611136 bytes 2022-04-10T01:32:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6100 IOs, 2668361728 bytes 2022-04-10T01:32:14.459 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6200 IOs, 2728977408 bytes 2022-04-10T01:32:14.603 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6300 IOs, 2756568064 bytes 2022-04-10T01:32:14.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6400 IOs, 2761521152 bytes 2022-04-10T01:32:14.856 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6500 IOs, 2790124544 bytes 2022-04-10T01:32:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6600 IOs, 2837217280 bytes 2022-04-10T01:32:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6700 IOs, 2857030656 bytes 2022-04-10T01:32:15.291 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6800 IOs, 2899979264 bytes 2022-04-10T01:32:15.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6900 IOs, 2979847168 bytes 2022-04-10T01:32:15.659 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7000 IOs, 3007870976 bytes 2022-04-10T01:32:15.795 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7100 IOs, 3028704256 bytes 2022-04-10T01:32:15.937 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7200 IOs, 3057967104 bytes 2022-04-10T01:32:16.074 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7300 IOs, 3082989568 bytes 2022-04-10T01:32:16.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7400 IOs, 3091111936 bytes 2022-04-10T01:32:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7500 IOs, 3131738112 bytes 2022-04-10T01:32:16.453 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:32:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:32:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:32:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:32:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:32:16.455 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_25 2022-04-10T01:32:16.455 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image_and_enable_mirror cluster2 mirror image_25 snapshot 128M 2022-04-10T01:32:16.455 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:32:16.455 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:32:16.456 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:32:16.456 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:32:16.456 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_25 2022-04-10T01:32:16.457 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:32:16.457 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:32:16.457 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:32:16.458 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:32:16.458 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:32:16.458 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:32:16.458 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snapshot ']' 2022-04-10T01:32:16.458 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:32:16.459 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_25 128M 2022-04-10T01:32:16.459 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:32:16.459 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:32:16.460 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:32:16.460 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:32:16.460 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_25 2022-04-10T01:32:16.460 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:32:16.461 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:32:16.461 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:32:16.461 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:32:16.461 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:32:16.462 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:32:16.462 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 128M ']' 2022-04-10T01:32:16.462 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=128M 2022-04-10T01:32:16.462 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:32:16.463 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_25 2022-04-10T01:32:16.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' image = image ']' 2022-04-10T01:32:16.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_25 snapshot 2022-04-10T01:32:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:32:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:32:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_25 2022-04-10T01:32:16.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:32:16.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:32:16.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:32:16.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:32:16.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:32:16.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_25 snapshot 2022-04-10T01:32:16.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:32:16.619 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_25 2022-04-10T01:32:16.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_25': 2022-04-10T01:32:16.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:32:16.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:32:16.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:32:16.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 133cb72ddc7b 2022-04-10T01:32:16.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.133cb72ddc7b 2022-04-10T01:32:16.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:32:16.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:32:16.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:32:16.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:32:16.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:32:16 2022 2022-04-10T01:32:16.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:32:16 2022 2022-04-10T01:32:16.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:32:16 2022 2022-04-10T01:32:16.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:32:16.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:32:16.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: a8a88faf-cd63-4557-9afd-ca8d6505edb6 2022-04-10T01:32:16.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:32:16.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:32:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_25 2022-04-10T01:32:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:32:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:32:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_25 2022-04-10T01:32:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:32:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=14 2022-04-10T01:32:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:32:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:32:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:32:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:32:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:32:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_25 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:32:16.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 25293824 bytes 2022-04-10T01:32:16.933 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 27348992 bytes 2022-04-10T01:32:17.065 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 66428928 bytes 2022-04-10T01:32:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 128007168 bytes 2022-04-10T01:32:17.389 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 157532160 bytes 2022-04-10T01:32:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 187660288 bytes 2022-04-10T01:32:17.733 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 260702208 bytes 2022-04-10T01:32:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 275901440 bytes 2022-04-10T01:32:17.977 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 290519040 bytes 2022-04-10T01:32:18.090 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 296655872 bytes 2022-04-10T01:32:18.252 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 344893440 bytes 2022-04-10T01:32:18.383 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 363055104 bytes 2022-04-10T01:32:18.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 426734592 bytes 2022-04-10T01:32:18.702 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 436226048 bytes 2022-04-10T01:32:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 436837376 bytes 2022-04-10T01:32:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 518508544 bytes 2022-04-10T01:32:19.180 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 588552192 bytes 2022-04-10T01:32:19.339 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 608779264 bytes 2022-04-10T01:32:19.467 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 633144320 bytes 2022-04-10T01:32:19.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 689772544 bytes 2022-04-10T01:32:19.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 740706304 bytes 2022-04-10T01:32:20.025 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 826387456 bytes 2022-04-10T01:32:20.169 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 852597760 bytes 2022-04-10T01:32:20.332 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 892772352 bytes 2022-04-10T01:32:20.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 993748992 bytes 2022-04-10T01:32:20.751 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1049785344 bytes 2022-04-10T01:32:20.945 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1124918272 bytes 2022-04-10T01:32:21.060 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1132982272 bytes 2022-04-10T01:32:21.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1188547584 bytes 2022-04-10T01:32:21.411 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1241781248 bytes 2022-04-10T01:32:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1269578752 bytes 2022-04-10T01:32:21.723 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1321473024 bytes 2022-04-10T01:32:21.866 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1352520704 bytes 2022-04-10T01:32:22.016 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1388839936 bytes 2022-04-10T01:32:22.150 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1409876992 bytes 2022-04-10T01:32:22.314 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1459497984 bytes 2022-04-10T01:32:22.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1544139776 bytes 2022-04-10T01:32:22.727 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1620070400 bytes 2022-04-10T01:32:22.897 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1666319360 bytes 2022-04-10T01:32:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1727210496 bytes 2022-04-10T01:32:23.243 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1775126528 bytes 2022-04-10T01:32:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 1809879040 bytes 2022-04-10T01:32:23.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 1866356736 bytes 2022-04-10T01:32:23.762 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 1935919104 bytes 2022-04-10T01:32:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 1946280960 bytes 2022-04-10T01:32:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 2024478720 bytes 2022-04-10T01:32:24.217 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 2047106048 bytes 2022-04-10T01:32:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 2074580992 bytes 2022-04-10T01:32:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 2113139712 bytes 2022-04-10T01:32:24.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 2144372736 bytes 2022-04-10T01:32:24.872 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 2233263104 bytes 2022-04-10T01:32:25.034 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 2281339904 bytes 2022-04-10T01:32:25.209 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 2338892800 bytes 2022-04-10T01:32:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5400 IOs, 2374103040 bytes 2022-04-10T01:32:25.541 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5500 IOs, 2424709120 bytes 2022-04-10T01:32:25.553 INFO:tasks.ceph.cluster2.osd.1.smithi043.stderr:2022-04-10T01:32:25.550+0000 7f18011ce700 -1 reset not still connected to 0x55b2d48f0270 2022-04-10T01:32:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5600 IOs, 2477677568 bytes 2022-04-10T01:32:25.847 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5700 IOs, 2500502528 bytes 2022-04-10T01:32:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5800 IOs, 2522815488 bytes 2022-04-10T01:32:26.097 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5900 IOs, 2529684480 bytes 2022-04-10T01:32:26.218 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6000 IOs, 2537520128 bytes 2022-04-10T01:32:26.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6100 IOs, 2596898816 bytes 2022-04-10T01:32:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6200 IOs, 2624303104 bytes 2022-04-10T01:32:26.648 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6300 IOs, 2627693568 bytes 2022-04-10T01:32:26.789 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6400 IOs, 2656700416 bytes 2022-04-10T01:32:26.979 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6500 IOs, 2720778240 bytes 2022-04-10T01:32:27.161 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6600 IOs, 2783585280 bytes 2022-04-10T01:32:27.347 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6700 IOs, 2863873024 bytes 2022-04-10T01:32:27.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6800 IOs, 2924193792 bytes 2022-04-10T01:32:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6900 IOs, 3011116032 bytes 2022-04-10T01:32:27.910 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7000 IOs, 3050285056 bytes 2022-04-10T01:32:28.077 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7100 IOs, 3127569408 bytes 2022-04-10T01:32:28.235 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7200 IOs, 3142167552 bytes 2022-04-10T01:32:28.386 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7300 IOs, 3176327168 bytes 2022-04-10T01:32:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7400 IOs, 3207252992 bytes 2022-04-10T01:32:28.725 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7500 IOs, 3269103616 bytes 2022-04-10T01:32:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7600 IOs, 3361928192 bytes 2022-04-10T01:32:29.123 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7700 IOs, 3413594112 bytes 2022-04-10T01:32:29.287 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7800 IOs, 3461882880 bytes 2022-04-10T01:32:29.419 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7900 IOs, 3491775488 bytes 2022-04-10T01:32:29.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8000 IOs, 3532450816 bytes 2022-04-10T01:32:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8100 IOs, 3549693952 bytes 2022-04-10T01:32:29.908 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8200 IOs, 3601133568 bytes 2022-04-10T01:32:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8300 IOs, 3628772352 bytes 2022-04-10T01:32:30.179 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8400 IOs, 3645385728 bytes 2022-04-10T01:32:30.319 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8500 IOs, 3679164416 bytes 2022-04-10T01:32:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8600 IOs, 3696411648 bytes 2022-04-10T01:32:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8700 IOs, 3703400448 bytes 2022-04-10T01:32:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:32:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:32:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:32:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:32:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:32:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_26 2022-04-10T01:32:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image_and_enable_mirror cluster2 mirror image_26 snapshot 128M 2022-04-10T01:32:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:32:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:32:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:32:30.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:32:30.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_26 2022-04-10T01:32:30.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:32:30.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:32:30.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:32:30.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:32:30.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:32:30.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:32:30.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snapshot ']' 2022-04-10T01:32:30.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:32:30.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_26 128M 2022-04-10T01:32:30.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:32:30.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:32:30.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:32:30.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:32:30.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_26 2022-04-10T01:32:30.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:32:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:32:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:32:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:32:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:32:30.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:32:30.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 128M ']' 2022-04-10T01:32:30.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=128M 2022-04-10T01:32:30.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:32:30.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_26 2022-04-10T01:32:30.705 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' image = image ']' 2022-04-10T01:32:30.706 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_26 snapshot 2022-04-10T01:32:30.706 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:32:30.706 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:32:30.706 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_26 2022-04-10T01:32:30.707 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:32:30.707 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:32:30.707 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:32:30.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:32:30.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:32:30.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_26 snapshot 2022-04-10T01:32:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:32:30.834 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_26 2022-04-10T01:32:30.867 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_26': 2022-04-10T01:32:30.868 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:32:30.869 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:32:30.869 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:32:30.869 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 1345113a053c 2022-04-10T01:32:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.1345113a053c 2022-04-10T01:32:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:32:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:32:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:32:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:32:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:32:30 2022 2022-04-10T01:32:30.872 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:32:30 2022 2022-04-10T01:32:30.872 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:32:30 2022 2022-04-10T01:32:30.872 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:32:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:32:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: ef9f36c8-5f7f-4c73-8f03-3933b20566c7 2022-04-10T01:32:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:32:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:32:30.877 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_26 2022-04-10T01:32:30.877 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:32:30.877 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:32:30.877 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_26 2022-04-10T01:32:30.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:32:30.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=13 2022-04-10T01:32:30.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:32:30.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:32:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:32:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:32:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:32:30.880 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_26 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:32:31.049 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 24100864 bytes 2022-04-10T01:32:31.233 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 104969216 bytes 2022-04-10T01:32:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 128491520 bytes 2022-04-10T01:32:31.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 143842304 bytes 2022-04-10T01:32:31.608 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 148405248 bytes 2022-04-10T01:32:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 182200320 bytes 2022-04-10T01:32:31.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 236486656 bytes 2022-04-10T01:32:32.072 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 260798464 bytes 2022-04-10T01:32:32.234 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 302182400 bytes 2022-04-10T01:32:32.395 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 355016704 bytes 2022-04-10T01:32:32.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 367022080 bytes 2022-04-10T01:32:32.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 427091968 bytes 2022-04-10T01:32:32.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 427704320 bytes 2022-04-10T01:32:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 455524352 bytes 2022-04-10T01:32:33.067 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 463257600 bytes 2022-04-10T01:32:33.215 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 496961536 bytes 2022-04-10T01:32:33.428 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 578455552 bytes 2022-04-10T01:32:33.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 606473216 bytes 2022-04-10T01:32:33.700 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 620773376 bytes 2022-04-10T01:32:33.883 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 682914816 bytes 2022-04-10T01:32:34.077 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 753522688 bytes 2022-04-10T01:32:34.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 804873216 bytes 2022-04-10T01:32:34.418 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 858752000 bytes 2022-04-10T01:32:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 919797760 bytes 2022-04-10T01:32:34.813 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 999980032 bytes 2022-04-10T01:32:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1075351552 bytes 2022-04-10T01:32:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1146139648 bytes 2022-04-10T01:32:35.377 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1186081792 bytes 2022-04-10T01:32:35.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1240836096 bytes 2022-04-10T01:32:35.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1276745728 bytes 2022-04-10T01:32:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1302066176 bytes 2022-04-10T01:32:35.964 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1324006400 bytes 2022-04-10T01:32:36.165 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1406725120 bytes 2022-04-10T01:32:36.374 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1485274112 bytes 2022-04-10T01:32:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1567937536 bytes 2022-04-10T01:32:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1622255616 bytes 2022-04-10T01:32:36.883 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1636921344 bytes 2022-04-10T01:32:37.027 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1669866496 bytes 2022-04-10T01:32:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1693224960 bytes 2022-04-10T01:32:37.329 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1735176192 bytes 2022-04-10T01:32:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1766254592 bytes 2022-04-10T01:32:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 1809257472 bytes 2022-04-10T01:32:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 1848733696 bytes 2022-04-10T01:32:37.925 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 1849371648 bytes 2022-04-10T01:32:38.045 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 1885498368 bytes 2022-04-10T01:32:38.217 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 1911769088 bytes 2022-04-10T01:32:38.362 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 1941255168 bytes 2022-04-10T01:32:38.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 1976671232 bytes 2022-04-10T01:32:38.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 2017214464 bytes 2022-04-10T01:32:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 2060234752 bytes 2022-04-10T01:32:39.007 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 2106125312 bytes 2022-04-10T01:32:39.173 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 2149756928 bytes 2022-04-10T01:32:39.317 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 2173515776 bytes 2022-04-10T01:32:39.448 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5400 IOs, 2180027392 bytes 2022-04-10T01:32:39.616 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5500 IOs, 2229664768 bytes 2022-04-10T01:32:39.781 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5600 IOs, 2287964160 bytes 2022-04-10T01:32:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5700 IOs, 2298615808 bytes 2022-04-10T01:32:40.169 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5800 IOs, 2387041280 bytes 2022-04-10T01:32:40.362 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5900 IOs, 2454725632 bytes 2022-04-10T01:32:40.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6000 IOs, 2538940416 bytes 2022-04-10T01:32:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6100 IOs, 2561947648 bytes 2022-04-10T01:32:40.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6200 IOs, 2572972032 bytes 2022-04-10T01:32:40.973 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6300 IOs, 2584194048 bytes 2022-04-10T01:32:41.129 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6400 IOs, 2621966336 bytes 2022-04-10T01:32:41.309 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6500 IOs, 2670157824 bytes 2022-04-10T01:32:41.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6600 IOs, 2718087168 bytes 2022-04-10T01:32:41.799 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6700 IOs, 2749087744 bytes 2022-04-10T01:32:41.965 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6800 IOs, 2777085952 bytes 2022-04-10T01:32:42.182 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6900 IOs, 2817855488 bytes 2022-04-10T01:32:42.341 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7000 IOs, 2856638464 bytes 2022-04-10T01:32:42.465 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7100 IOs, 2882938880 bytes 2022-04-10T01:32:42.681 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7200 IOs, 2955893760 bytes 2022-04-10T01:32:42.844 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7300 IOs, 2997694464 bytes 2022-04-10T01:32:43.093 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7400 IOs, 3118608384 bytes 2022-04-10T01:32:43.255 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7500 IOs, 3158111232 bytes 2022-04-10T01:32:43.431 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7600 IOs, 3209930752 bytes 2022-04-10T01:32:43.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7700 IOs, 3247891456 bytes 2022-04-10T01:32:43.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7800 IOs, 3346722816 bytes 2022-04-10T01:32:43.881 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:32:43.881 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:32:43.881 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:32:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:32:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:32:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_27 2022-04-10T01:32:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image_and_enable_mirror cluster2 mirror image_27 snapshot 128M 2022-04-10T01:32:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:32:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:32:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:32:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:32:43.884 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_27 2022-04-10T01:32:43.884 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:32:43.884 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:32:43.884 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:32:43.884 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:32:43.885 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:32:43.885 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:32:43.885 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snapshot ']' 2022-04-10T01:32:43.886 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:32:43.886 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_27 128M 2022-04-10T01:32:43.886 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:32:43.886 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:32:43.886 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:32:43.887 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:32:43.887 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_27 2022-04-10T01:32:43.887 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:32:43.887 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:32:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:32:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:32:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:32:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:32:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 128M ']' 2022-04-10T01:32:43.889 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=128M 2022-04-10T01:32:43.889 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:32:43.889 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_27 2022-04-10T01:32:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' image = image ']' 2022-04-10T01:32:43.926 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_27 snapshot 2022-04-10T01:32:43.926 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:32:43.926 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:32:43.926 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_27 2022-04-10T01:32:43.927 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:32:43.927 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:32:43.927 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:32:43.927 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:32:43.927 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:32:43.928 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_27 snapshot 2022-04-10T01:32:44.403 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:32:44.412 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_27 2022-04-10T01:32:44.447 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_27': 2022-04-10T01:32:44.447 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:32:44.447 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:32:44.447 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:32:44.448 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 134d16b178fd 2022-04-10T01:32:44.448 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.134d16b178fd 2022-04-10T01:32:44.448 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:32:44.448 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:32:44.448 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:32:44.449 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:32:44.449 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:32:43 2022 2022-04-10T01:32:44.449 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:32:43 2022 2022-04-10T01:32:44.449 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:32:43 2022 2022-04-10T01:32:44.449 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:32:44.450 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:32:44.450 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 5e030d77-6c1f-4e4a-a414-67687916c7e7 2022-04-10T01:32:44.450 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:32:44.452 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:32:44.452 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_27 2022-04-10T01:32:44.452 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:32:44.452 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:32:44.453 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_27 2022-04-10T01:32:44.453 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:32:44.453 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=9 2022-04-10T01:32:44.453 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:32:44.453 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:32:44.454 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:32:44.454 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:32:44.454 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:32:44.454 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_27 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:32:44.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 64026624 bytes 2022-04-10T01:32:44.780 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 64674816 bytes 2022-04-10T01:32:44.930 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 96466944 bytes 2022-04-10T01:32:45.069 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 118644736 bytes 2022-04-10T01:32:45.232 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 167693312 bytes 2022-04-10T01:32:45.371 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 202825728 bytes 2022-04-10T01:32:45.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 239849472 bytes 2022-04-10T01:32:45.704 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 293580800 bytes 2022-04-10T01:32:45.886 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 361794560 bytes 2022-04-10T01:32:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 413872128 bytes 2022-04-10T01:32:46.178 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 429623296 bytes 2022-04-10T01:32:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 452844544 bytes 2022-04-10T01:32:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 477104128 bytes 2022-04-10T01:32:46.609 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 514280448 bytes 2022-04-10T01:32:46.753 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 551374848 bytes 2022-04-10T01:32:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 615191552 bytes 2022-04-10T01:32:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 667409408 bytes 2022-04-10T01:32:47.474 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 710637568 bytes 2022-04-10T01:32:47.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 770401280 bytes 2022-04-10T01:32:47.823 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 811321344 bytes 2022-04-10T01:32:47.988 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 858984448 bytes 2022-04-10T01:32:48.136 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 891851776 bytes 2022-04-10T01:32:48.261 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 916032512 bytes 2022-04-10T01:32:48.459 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 974668800 bytes 2022-04-10T01:32:48.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 1022492672 bytes 2022-04-10T01:32:48.755 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1043957760 bytes 2022-04-10T01:32:48.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1103480832 bytes 2022-04-10T01:32:49.167 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1191484416 bytes 2022-04-10T01:32:49.304 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1214265344 bytes 2022-04-10T01:32:49.444 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1238136832 bytes 2022-04-10T01:32:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1297518592 bytes 2022-04-10T01:32:49.788 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1336160256 bytes 2022-04-10T01:32:49.834 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr:2022-04-10T01:32:49.831+0000 7f52fc8d6700 -1 reset not still connected to 0x5600f3dbe0d0 2022-04-10T01:32:49.901 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1341549568 bytes 2022-04-10T01:32:50.076 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1399120896 bytes 2022-04-10T01:32:50.231 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1433448448 bytes 2022-04-10T01:32:50.339 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1434064896 bytes 2022-04-10T01:32:50.544 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1512929280 bytes 2022-04-10T01:32:50.714 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1562112000 bytes 2022-04-10T01:32:50.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1589006336 bytes 2022-04-10T01:32:51.044 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1649177600 bytes 2022-04-10T01:32:51.174 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1665764352 bytes 2022-04-10T01:32:51.382 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 1744508928 bytes 2022-04-10T01:32:51.586 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 1820836864 bytes 2022-04-10T01:32:51.720 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 1840113664 bytes 2022-04-10T01:32:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 1858563072 bytes 2022-04-10T01:32:52.011 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 1897230336 bytes 2022-04-10T01:32:52.180 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 1942296576 bytes 2022-04-10T01:32:52.318 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 1963152384 bytes 2022-04-10T01:32:52.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 2014018560 bytes 2022-04-10T01:32:52.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 2067254272 bytes 2022-04-10T01:32:52.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 2099544064 bytes 2022-04-10T01:32:52.951 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 2132729856 bytes 2022-04-10T01:32:53.061 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 2133377024 bytes 2022-04-10T01:32:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5400 IOs, 2189972480 bytes 2022-04-10T01:32:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5500 IOs, 2216199168 bytes 2022-04-10T01:32:53.459 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:32:53.459 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:32:53.459 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:32:53.460 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:32:53.460 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:32:53.460 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_28 2022-04-10T01:32:53.460 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image_and_enable_mirror cluster2 mirror image_28 snapshot 128M 2022-04-10T01:32:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:32:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:32:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:32:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:32:53.462 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_28 2022-04-10T01:32:53.462 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:32:53.462 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:32:53.462 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:32:53.463 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:32:53.463 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:32:53.463 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:32:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snapshot ']' 2022-04-10T01:32:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:32:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_28 128M 2022-04-10T01:32:53.466 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:32:53.466 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:32:53.466 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:32:53.466 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:32:53.466 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_28 2022-04-10T01:32:53.467 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:32:53.467 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:32:53.467 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:32:53.467 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:32:53.468 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:32:53.468 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:32:53.468 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 128M ']' 2022-04-10T01:32:53.468 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=128M 2022-04-10T01:32:53.468 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:32:53.469 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_28 2022-04-10T01:32:53.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' image = image ']' 2022-04-10T01:32:53.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_28 snapshot 2022-04-10T01:32:53.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:32:53.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:32:53.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_28 2022-04-10T01:32:53.505 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:32:53.505 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:32:53.505 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:32:53.505 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:32:53.506 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:32:53.506 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_28 snapshot 2022-04-10T01:32:54.413 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:32:54.425 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_28 2022-04-10T01:32:54.461 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_28': 2022-04-10T01:32:54.462 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:32:54.462 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:32:54.462 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:32:54.462 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 1356175e8644 2022-04-10T01:32:54.462 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.1356175e8644 2022-04-10T01:32:54.463 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:32:54.463 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:32:54.463 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:32:54.463 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:32:54.464 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:32:53 2022 2022-04-10T01:32:54.464 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:32:53 2022 2022-04-10T01:32:54.464 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:32:53 2022 2022-04-10T01:32:54.464 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:32:54.465 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:32:54.465 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: e100174d-382f-42fc-9a30-2ef4065b1b6a 2022-04-10T01:32:54.465 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:32:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:32:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_28 2022-04-10T01:32:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:32:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:32:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_28 2022-04-10T01:32:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:32:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=13 2022-04-10T01:32:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:32:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:32:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:32:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:32:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:32:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_28 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:32:54.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 48013312 bytes 2022-04-10T01:32:54.838 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 95693824 bytes 2022-04-10T01:32:54.982 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 131619840 bytes 2022-04-10T01:32:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 132242432 bytes 2022-04-10T01:32:55.242 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 176632832 bytes 2022-04-10T01:32:55.423 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 224707584 bytes 2022-04-10T01:32:55.566 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 250408960 bytes 2022-04-10T01:32:55.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 272361472 bytes 2022-04-10T01:32:55.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 302156800 bytes 2022-04-10T01:32:56.041 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 385888256 bytes 2022-04-10T01:32:56.293 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 494649344 bytes 2022-04-10T01:32:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 540493824 bytes 2022-04-10T01:32:56.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 616659968 bytes 2022-04-10T01:32:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 662454272 bytes 2022-04-10T01:32:56.962 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 670996480 bytes 2022-04-10T01:32:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 718166016 bytes 2022-04-10T01:32:57.300 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 760622080 bytes 2022-04-10T01:32:57.407 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 761241600 bytes 2022-04-10T01:32:57.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 794800128 bytes 2022-04-10T01:32:57.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 843138048 bytes 2022-04-10T01:32:57.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 942780416 bytes 2022-04-10T01:32:58.126 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 1007923200 bytes 2022-04-10T01:32:58.255 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 1008524288 bytes 2022-04-10T01:32:58.410 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 1047126016 bytes 2022-04-10T01:32:58.589 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 1120120832 bytes 2022-04-10T01:32:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1177347072 bytes 2022-04-10T01:32:58.916 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1180318720 bytes 2022-04-10T01:32:59.026 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5620211946798548) > weight (0.25) 2022-04-10T01:32:59.027 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.6557404158756601) > weight (0.25) 2022-04-10T01:32:59.027 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2022-04-10T01:32:59.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2022-04-10T01:32:59.028 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2022-04-10T01:32:59.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi043.stderr:2022-04-10T01:32:59.026+0000 7f1491ff0700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 43952) UID: 1000 2022-04-10T01:32:59.081 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1212211200 bytes 2022-04-10T01:32:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1224824832 bytes 2022-04-10T01:32:59.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi043.stderr:daemon-helper: command failed with exit status 1 2022-04-10T01:32:59.376 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1267121152 bytes 2022-04-10T01:32:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1327633408 bytes 2022-04-10T01:32:59.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1366728704 bytes 2022-04-10T01:32:59.849 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1377362944 bytes 2022-04-10T01:33:00.006 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1412823040 bytes 2022-04-10T01:33:00.159 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1441605632 bytes 2022-04-10T01:33:00.294 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1462594560 bytes 2022-04-10T01:33:00.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1523091456 bytes 2022-04-10T01:33:00.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1542591488 bytes 2022-04-10T01:33:00.854 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1646977024 bytes 2022-04-10T01:33:01.030 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1701065728 bytes 2022-04-10T01:33:01.180 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1730109440 bytes 2022-04-10T01:33:01.327 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 1757733888 bytes 2022-04-10T01:33:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 1856537600 bytes 2022-04-10T01:33:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 1890134016 bytes 2022-04-10T01:33:01.847 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 1935197184 bytes 2022-04-10T01:33:01.968 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 1947698176 bytes 2022-04-10T01:33:02.029 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2022-04-10T01:33:02.030 INFO:teuthology.orchestra.run:waiting for 600 2022-04-10T01:33:02.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-04-10T01:33:02.031 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2022-04-10T01:33:02.031 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2022-04-10T01:33:02.032 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2022-04-10T01:33:02.040 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2022-04-10T01:33:02.040 INFO:tasks.rbd_mirror_thrash:waiting for 103 secs before thrashing 2022-04-10T01:33:02.095 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 1957344256 bytes 2022-04-10T01:33:02.256 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 1980800000 bytes 2022-04-10T01:33:02.396 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 1981417472 bytes 2022-04-10T01:33:02.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 2025654272 bytes 2022-04-10T01:33:02.700 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 2050778112 bytes 2022-04-10T01:33:02.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 2093219840 bytes 2022-04-10T01:33:03.020 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 2135965696 bytes 2022-04-10T01:33:03.214 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5400 IOs, 2210665472 bytes 2022-04-10T01:33:03.406 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5500 IOs, 2275985408 bytes 2022-04-10T01:33:03.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5600 IOs, 2283170816 bytes 2022-04-10T01:33:03.733 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5700 IOs, 2374162432 bytes 2022-04-10T01:33:03.911 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5800 IOs, 2435628032 bytes 2022-04-10T01:33:04.043 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5900 IOs, 2458179584 bytes 2022-04-10T01:33:04.154 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6000 IOs, 2460273664 bytes 2022-04-10T01:33:04.316 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6100 IOs, 2495157248 bytes 2022-04-10T01:33:04.495 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6200 IOs, 2555310080 bytes 2022-04-10T01:33:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6300 IOs, 2568875008 bytes 2022-04-10T01:33:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6400 IOs, 2653234176 bytes 2022-04-10T01:33:04.982 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6500 IOs, 2695674880 bytes 2022-04-10T01:33:05.155 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6600 IOs, 2749584384 bytes 2022-04-10T01:33:05.352 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6700 IOs, 2820927488 bytes 2022-04-10T01:33:05.506 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6800 IOs, 2861452288 bytes 2022-04-10T01:33:05.649 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6900 IOs, 2904188928 bytes 2022-04-10T01:33:05.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7000 IOs, 2940513280 bytes 2022-04-10T01:33:05.972 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7100 IOs, 2981019648 bytes 2022-04-10T01:33:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7200 IOs, 3005498368 bytes 2022-04-10T01:33:06.270 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7300 IOs, 3059145728 bytes 2022-04-10T01:33:06.439 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7400 IOs, 3101788160 bytes 2022-04-10T01:33:06.614 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7500 IOs, 3160765440 bytes 2022-04-10T01:33:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7600 IOs, 3175741440 bytes 2022-04-10T01:33:06.892 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7700 IOs, 3213831168 bytes 2022-04-10T01:33:07.040 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7800 IOs, 3252277248 bytes 2022-04-10T01:33:07.180 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7900 IOs, 3278246912 bytes 2022-04-10T01:33:07.308 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8000 IOs, 3289407488 bytes 2022-04-10T01:33:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:33:07.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:33:07.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:33:07.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:33:07.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:33:07.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_29 2022-04-10T01:33:07.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image_and_enable_mirror cluster2 mirror image_29 snapshot 128M 2022-04-10T01:33:07.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:33:07.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:33:07.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:33:07.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:33:07.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_29 2022-04-10T01:33:07.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:33:07.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:33:07.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:33:07.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:33:07.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:33:07.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:33:07.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snapshot ']' 2022-04-10T01:33:07.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:33:07.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_29 128M 2022-04-10T01:33:07.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:33:07.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:33:07.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:33:07.483 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:33:07.483 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_29 2022-04-10T01:33:07.483 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:33:07.483 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:33:07.484 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:33:07.484 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:33:07.484 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:33:07.484 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:33:07.484 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 128M ']' 2022-04-10T01:33:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=128M 2022-04-10T01:33:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:33:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_29 2022-04-10T01:33:07.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' image = image ']' 2022-04-10T01:33:07.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_29 snapshot 2022-04-10T01:33:07.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:33:07.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:33:07.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_29 2022-04-10T01:33:07.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:33:07.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:33:07.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:33:07.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:33:07.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:33:07.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_29 snapshot 2022-04-10T01:33:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:33:08.424 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_29 2022-04-10T01:33:08.460 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_29': 2022-04-10T01:33:08.461 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:33:08.461 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:33:08.461 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:33:08.462 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 13693cc896f4 2022-04-10T01:33:08.462 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.13693cc896f4 2022-04-10T01:33:08.462 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:33:08.462 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:33:08.463 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:33:08.463 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:33:08.463 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:33:07 2022 2022-04-10T01:33:08.463 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:33:07 2022 2022-04-10T01:33:08.464 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:33:07 2022 2022-04-10T01:33:08.464 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:33:08.464 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:33:08.465 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 617e30a5-cc17-4271-95fb-ba4832408157 2022-04-10T01:33:08.465 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:33:08.467 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:33:08.467 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_29 2022-04-10T01:33:08.467 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:33:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:33:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_29 2022-04-10T01:33:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:33:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=12 2022-04-10T01:33:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:33:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:33:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:33:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:33:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:33:08.470 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_29 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:33:08.648 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 34268160 bytes 2022-04-10T01:33:08.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 81043456 bytes 2022-04-10T01:33:08.922 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 87703552 bytes 2022-04-10T01:33:09.099 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 147251200 bytes 2022-04-10T01:33:09.224 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 157356032 bytes 2022-04-10T01:33:09.374 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 206323712 bytes 2022-04-10T01:33:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 254681088 bytes 2022-04-10T01:33:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 307972096 bytes 2022-04-10T01:33:09.964 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 403815424 bytes 2022-04-10T01:33:10.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 465649664 bytes 2022-04-10T01:33:10.388 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 532488192 bytes 2022-04-10T01:33:10.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 602917888 bytes 2022-04-10T01:33:10.763 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 652731392 bytes 2022-04-10T01:33:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 696400896 bytes 2022-04-10T01:33:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 703709184 bytes 2022-04-10T01:33:11.225 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 741492736 bytes 2022-04-10T01:33:11.362 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 751053824 bytes 2022-04-10T01:33:11.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 810967040 bytes 2022-04-10T01:33:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 844698624 bytes 2022-04-10T01:33:12.040 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 918583296 bytes 2022-04-10T01:33:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 954660864 bytes 2022-04-10T01:33:12.326 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 968942592 bytes 2022-04-10T01:33:12.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 1037469696 bytes 2022-04-10T01:33:12.783 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 1160031232 bytes 2022-04-10T01:33:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 1204793344 bytes 2022-04-10T01:33:13.122 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1257497600 bytes 2022-04-10T01:33:13.286 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1310163968 bytes 2022-04-10T01:33:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1366976512 bytes 2022-04-10T01:33:13.607 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1384736768 bytes 2022-04-10T01:33:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1409666048 bytes 2022-04-10T01:33:13.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1430230016 bytes 2022-04-10T01:33:14.072 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1498317824 bytes 2022-04-10T01:33:14.276 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1580729344 bytes 2022-04-10T01:33:14.408 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1589938176 bytes 2022-04-10T01:33:14.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1639127040 bytes 2022-04-10T01:33:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1685324800 bytes 2022-04-10T01:33:14.882 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1711438848 bytes 2022-04-10T01:33:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1718823936 bytes 2022-04-10T01:33:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1753428992 bytes 2022-04-10T01:33:15.305 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1793330176 bytes 2022-04-10T01:33:15.487 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1844476928 bytes 2022-04-10T01:33:15.650 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 1893566464 bytes 2022-04-10T01:33:15.772 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 1906110464 bytes 2022-04-10T01:33:15.911 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 1931149312 bytes 2022-04-10T01:33:16.064 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 1964839936 bytes 2022-04-10T01:33:16.229 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 2007703552 bytes 2022-04-10T01:33:16.348 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 2013982720 bytes 2022-04-10T01:33:16.483 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 2054484992 bytes 2022-04-10T01:33:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 2144531456 bytes 2022-04-10T01:33:16.830 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6307049848857736) > weight (0.25) 2022-04-10T01:33:16.831 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.365205809010135) > weight (0.25) 2022-04-10T01:33:16.831 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.3060040514393766) > weight (0.25) 2022-04-10T01:33:16.831 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.43009423782720224) > weight (0.25) 2022-04-10T01:33:16.832 INFO:tasks.rbd_mirror_thrash:waiting for 74 secs before thrashing 2022-04-10T01:33:16.898 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 2197788672 bytes 2022-04-10T01:33:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 2279827456 bytes 2022-04-10T01:33:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 2293108736 bytes 2022-04-10T01:33:17.432 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 2371055616 bytes 2022-04-10T01:33:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5400 IOs, 2438099968 bytes 2022-04-10T01:33:17.810 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5500 IOs, 2489972736 bytes 2022-04-10T01:33:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5600 IOs, 2602953728 bytes 2022-04-10T01:33:18.227 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5700 IOs, 2660507648 bytes 2022-04-10T01:33:18.335 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5800 IOs, 2661128192 bytes 2022-04-10T01:33:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5900 IOs, 2680270848 bytes 2022-04-10T01:33:18.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6000 IOs, 2718519296 bytes 2022-04-10T01:33:18.787 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6100 IOs, 2784252928 bytes 2022-04-10T01:33:18.949 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6200 IOs, 2834304000 bytes 2022-04-10T01:33:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6300 IOs, 2909523968 bytes 2022-04-10T01:33:19.289 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6400 IOs, 2961633280 bytes 2022-04-10T01:33:19.474 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6500 IOs, 3000536064 bytes 2022-04-10T01:33:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6600 IOs, 3084041216 bytes 2022-04-10T01:33:19.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6700 IOs, 3128995840 bytes 2022-04-10T01:33:19.960 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6800 IOs, 3142479872 bytes 2022-04-10T01:33:20.105 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6900 IOs, 3176064000 bytes 2022-04-10T01:33:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7000 IOs, 3200914432 bytes 2022-04-10T01:33:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7100 IOs, 3237977088 bytes 2022-04-10T01:33:20.474 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:33:20.474 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:33:20.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:33:20.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:33:20.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:33:20.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_30 2022-04-10T01:33:20.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image_and_enable_mirror cluster2 mirror image_30 snapshot 128M 2022-04-10T01:33:20.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:33:20.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:33:20.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:33:20.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:33:20.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_30 2022-04-10T01:33:20.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:33:20.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:33:20.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:33:20.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:33:20.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:33:20.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:33:20.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snapshot ']' 2022-04-10T01:33:20.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:33:20.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_30 128M 2022-04-10T01:33:20.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:33:20.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:33:20.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:33:20.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:33:20.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_30 2022-04-10T01:33:20.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:33:20.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:33:20.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:33:20.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:33:20.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:33:20.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:33:20.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 128M ']' 2022-04-10T01:33:20.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=128M 2022-04-10T01:33:20.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:33:20.483 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_30 2022-04-10T01:33:20.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' image = image ']' 2022-04-10T01:33:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_30 snapshot 2022-04-10T01:33:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:33:20.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:33:20.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_30 2022-04-10T01:33:20.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:33:20.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:33:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:33:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:33:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:33:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_30 snapshot 2022-04-10T01:33:20.632 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:33:20.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_30 2022-04-10T01:33:20.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_30': 2022-04-10T01:33:20.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:33:20.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:33:20.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:33:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 137241ff2172 2022-04-10T01:33:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.137241ff2172 2022-04-10T01:33:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:33:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:33:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:33:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:33:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:33:20 2022 2022-04-10T01:33:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:33:20 2022 2022-04-10T01:33:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:33:20 2022 2022-04-10T01:33:20.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:33:20.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:33:20.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: e756a8db-50ef-4000-ba87-3241379f1d82 2022-04-10T01:33:20.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:33:20.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:33:20.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_30 2022-04-10T01:33:20.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:33:20.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:33:20.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_30 2022-04-10T01:33:20.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:33:20.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=13 2022-04-10T01:33:20.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:33:20.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:33:20.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:33:20.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:33:20.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:33:20.681 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_30 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:33:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 28813312 bytes 2022-04-10T01:33:20.999 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 74214400 bytes 2022-04-10T01:33:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 96284672 bytes 2022-04-10T01:33:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 193954816 bytes 2022-04-10T01:33:21.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 258396160 bytes 2022-04-10T01:33:21.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 259005440 bytes 2022-04-10T01:33:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 277291008 bytes 2022-04-10T01:33:21.899 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 307177472 bytes 2022-04-10T01:33:22.049 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 352241664 bytes 2022-04-10T01:33:22.212 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 402503680 bytes 2022-04-10T01:33:22.376 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 456771584 bytes 2022-04-10T01:33:22.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 506173440 bytes 2022-04-10T01:33:22.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 537007104 bytes 2022-04-10T01:33:22.825 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 570525696 bytes 2022-04-10T01:33:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 594304000 bytes 2022-04-10T01:33:23.108 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 627058688 bytes 2022-04-10T01:33:23.221 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 638508032 bytes 2022-04-10T01:33:23.349 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 658846720 bytes 2022-04-10T01:33:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 689420288 bytes 2022-04-10T01:33:23.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 748434432 bytes 2022-04-10T01:33:23.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 769090560 bytes 2022-04-10T01:33:23.996 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 847085568 bytes 2022-04-10T01:33:24.128 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 859338752 bytes 2022-04-10T01:33:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 911109120 bytes 2022-04-10T01:33:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 927404032 bytes 2022-04-10T01:33:24.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 944779264 bytes 2022-04-10T01:33:24.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 959054848 bytes 2022-04-10T01:33:24.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 998734848 bytes 2022-04-10T01:33:24.949 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1022768128 bytes 2022-04-10T01:33:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1029206016 bytes 2022-04-10T01:33:25.186 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1050293248 bytes 2022-04-10T01:33:25.325 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1079201792 bytes 2022-04-10T01:33:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1087519744 bytes 2022-04-10T01:33:25.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1187886080 bytes 2022-04-10T01:33:25.833 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1238324224 bytes 2022-04-10T01:33:25.983 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1279174656 bytes 2022-04-10T01:33:26.116 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1306020864 bytes 2022-04-10T01:33:26.249 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1321780224 bytes 2022-04-10T01:33:26.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1354624000 bytes 2022-04-10T01:33:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1395571712 bytes 2022-04-10T01:33:26.750 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1468329984 bytes 2022-04-10T01:33:26.886 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 1495740416 bytes 2022-04-10T01:33:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 1518801920 bytes 2022-04-10T01:33:27.135 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 1535794176 bytes 2022-04-10T01:33:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 1552037888 bytes 2022-04-10T01:33:27.396 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 1576221696 bytes 2022-04-10T01:33:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 1672180736 bytes 2022-04-10T01:33:27.782 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 1715750912 bytes 2022-04-10T01:33:27.927 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 1746177024 bytes 2022-04-10T01:33:28.061 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 1767056384 bytes 2022-04-10T01:33:28.215 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 1805189120 bytes 2022-04-10T01:33:28.322 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 1805803520 bytes 2022-04-10T01:33:28.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 1891018752 bytes 2022-04-10T01:33:28.703 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5400 IOs, 1930901504 bytes 2022-04-10T01:33:28.843 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5500 IOs, 1960456192 bytes 2022-04-10T01:33:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5600 IOs, 2013857792 bytes 2022-04-10T01:33:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5700 IOs, 2021429248 bytes 2022-04-10T01:33:29.268 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5800 IOs, 2057346048 bytes 2022-04-10T01:33:29.389 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5900 IOs, 2074346496 bytes 2022-04-10T01:33:29.529 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6000 IOs, 2099368960 bytes 2022-04-10T01:33:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6100 IOs, 2107177984 bytes 2022-04-10T01:33:29.774 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6200 IOs, 2125870080 bytes 2022-04-10T01:33:29.915 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6300 IOs, 2150828032 bytes 2022-04-10T01:33:30.026 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6400 IOs, 2158441472 bytes 2022-04-10T01:33:30.234 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6500 IOs, 2260544512 bytes 2022-04-10T01:33:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6600 IOs, 2277070848 bytes 2022-04-10T01:33:30.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6700 IOs, 2303947776 bytes 2022-04-10T01:33:30.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6800 IOs, 2327323648 bytes 2022-04-10T01:33:30.854 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6900 IOs, 2411184128 bytes 2022-04-10T01:33:31.009 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7000 IOs, 2457427968 bytes 2022-04-10T01:33:31.143 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7100 IOs, 2472750080 bytes 2022-04-10T01:33:31.315 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7200 IOs, 2532663296 bytes 2022-04-10T01:33:31.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7300 IOs, 2604894208 bytes 2022-04-10T01:33:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7400 IOs, 2661988352 bytes 2022-04-10T01:33:31.847 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7500 IOs, 2704705536 bytes 2022-04-10T01:33:31.975 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7600 IOs, 2711162880 bytes 2022-04-10T01:33:32.127 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7700 IOs, 2745155584 bytes 2022-04-10T01:33:32.282 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7800 IOs, 2780514304 bytes 2022-04-10T01:33:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7900 IOs, 2848971776 bytes 2022-04-10T01:33:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8000 IOs, 2876275712 bytes 2022-04-10T01:33:32.723 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8100 IOs, 2876882944 bytes 2022-04-10T01:33:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8200 IOs, 2903999488 bytes 2022-04-10T01:33:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8300 IOs, 2979095552 bytes 2022-04-10T01:33:33.374 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8400 IOs, 2988360704 bytes 2022-04-10T01:33:33.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8500 IOs, 2988993536 bytes 2022-04-10T01:33:33.609 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8600 IOs, 3014309888 bytes 2022-04-10T01:33:33.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:33:33.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:33:33.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:33:33.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:33:33.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:33:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_31 2022-04-10T01:33:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image_and_enable_mirror cluster2 mirror image_31 snapshot 128M 2022-04-10T01:33:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:33:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:33:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:33:33.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:33:33.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_31 2022-04-10T01:33:33.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:33:33.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:33:33.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:33:33.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:33:33.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:33:33.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:33:33.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snapshot ']' 2022-04-10T01:33:33.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:33:33.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_31 128M 2022-04-10T01:33:33.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:33:33.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:33:33.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:33:33.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:33:33.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_31 2022-04-10T01:33:33.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:33:33.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:33:33.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:33:33.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:33:33.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:33:33.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:33:33.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 128M ']' 2022-04-10T01:33:33.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=128M 2022-04-10T01:33:33.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:33:33.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_31 2022-04-10T01:33:33.731 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' image = image ']' 2022-04-10T01:33:33.732 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_31 snapshot 2022-04-10T01:33:33.732 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:33:33.732 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:33:33.733 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_31 2022-04-10T01:33:33.733 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:33:33.734 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:33:33.734 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:33:33.735 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:33:33.735 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:33:33.735 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_31 snapshot 2022-04-10T01:33:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:33:34.589 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_31 2022-04-10T01:33:34.626 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_31': 2022-04-10T01:33:34.627 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:33:34.627 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:33:34.627 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:33:34.627 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 137ac601e798 2022-04-10T01:33:34.628 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.137ac601e798 2022-04-10T01:33:34.628 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:33:34.628 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:33:34.628 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:33:34.629 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:33:34.629 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:33:33 2022 2022-04-10T01:33:34.629 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:33:33 2022 2022-04-10T01:33:34.629 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:33:33 2022 2022-04-10T01:33:34.630 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:33:34.630 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:33:34.630 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 9c9c599b-a61d-4229-a0b1-ba391a8116f1 2022-04-10T01:33:34.630 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:33:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:33:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_31 2022-04-10T01:33:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:33:34.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:33:34.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_31 2022-04-10T01:33:34.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:33:34.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=10 2022-04-10T01:33:34.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:33:34.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:33:34.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:33:34.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:33:34.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:33:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_31 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:33:34.844 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 55705600 bytes 2022-04-10T01:33:34.980 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 85582848 bytes 2022-04-10T01:33:35.147 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 150059008 bytes 2022-04-10T01:33:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 156768256 bytes 2022-04-10T01:33:35.420 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 200431616 bytes 2022-04-10T01:33:35.605 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 264870912 bytes 2022-04-10T01:33:35.778 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 314587136 bytes 2022-04-10T01:33:35.898 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 326621184 bytes 2022-04-10T01:33:36.031 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 346779648 bytes 2022-04-10T01:33:36.185 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 386115584 bytes 2022-04-10T01:33:36.347 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 429881344 bytes 2022-04-10T01:33:36.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 486663168 bytes 2022-04-10T01:33:36.731 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 580471808 bytes 2022-04-10T01:33:36.996 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 693038080 bytes 2022-04-10T01:33:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 714612736 bytes 2022-04-10T01:33:37.299 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 763392000 bytes 2022-04-10T01:33:37.468 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 810175488 bytes 2022-04-10T01:33:37.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 861644800 bytes 2022-04-10T01:33:37.785 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 892590080 bytes 2022-04-10T01:33:37.921 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 921819136 bytes 2022-04-10T01:33:38.079 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 961246208 bytes 2022-04-10T01:33:38.259 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 1018146816 bytes 2022-04-10T01:33:38.409 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 1050416128 bytes 2022-04-10T01:33:38.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 1066449920 bytes 2022-04-10T01:33:38.701 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 1132453888 bytes 2022-04-10T01:33:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1169726464 bytes 2022-04-10T01:33:39.025 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1204030464 bytes 2022-04-10T01:33:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1295057920 bytes 2022-04-10T01:33:39.394 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1325396992 bytes 2022-04-10T01:33:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1385082880 bytes 2022-04-10T01:33:39.831 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1512461312 bytes 2022-04-10T01:33:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1516483584 bytes 2022-04-10T01:33:40.105 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1563627520 bytes 2022-04-10T01:33:40.270 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1605952512 bytes 2022-04-10T01:33:40.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1666205696 bytes 2022-04-10T01:33:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1686144000 bytes 2022-04-10T01:33:40.809 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1740737536 bytes 2022-04-10T01:33:41.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1816527872 bytes 2022-04-10T01:33:41.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1882162176 bytes 2022-04-10T01:33:41.799 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1890246656 bytes 2022-04-10T01:33:41.927 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1890889728 bytes 2022-04-10T01:33:42.075 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 1910282240 bytes 2022-04-10T01:33:42.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 1922683904 bytes 2022-04-10T01:33:42.397 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 1980898304 bytes 2022-04-10T01:33:42.543 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 2002225152 bytes 2022-04-10T01:33:42.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 2036025344 bytes 2022-04-10T01:33:42.969 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 2147720192 bytes 2022-04-10T01:33:43.246 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 2280309760 bytes 2022-04-10T01:33:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 2316608512 bytes 2022-04-10T01:33:43.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 2317217792 bytes 2022-04-10T01:33:43.681 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 2382695424 bytes 2022-04-10T01:33:43.839 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 2425249792 bytes 2022-04-10T01:33:43.982 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 2452451328 bytes 2022-04-10T01:33:44.129 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5400 IOs, 2507586560 bytes 2022-04-10T01:33:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5500 IOs, 2520275968 bytes 2022-04-10T01:33:44.419 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5600 IOs, 2555509760 bytes 2022-04-10T01:33:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5700 IOs, 2576635904 bytes 2022-04-10T01:33:44.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:33:44.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:33:44.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:33:44.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:33:44.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:33:44.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_32 2022-04-10T01:33:44.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image_and_enable_mirror cluster2 mirror image_32 snapshot 128M 2022-04-10T01:33:44.647 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:33:44.647 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:33:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:33:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:33:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_32 2022-04-10T01:33:44.649 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:33:44.649 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:33:44.649 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:33:44.650 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:33:44.650 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:33:44.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:33:44.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snapshot ']' 2022-04-10T01:33:44.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:33:44.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_32 128M 2022-04-10T01:33:44.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:33:44.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:33:44.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:33:44.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:33:44.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_32 2022-04-10T01:33:44.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:33:44.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:33:44.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:33:44.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:33:44.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:33:44.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:33:44.658 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 128M ']' 2022-04-10T01:33:44.658 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=128M 2022-04-10T01:33:44.658 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:33:44.659 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_32 2022-04-10T01:33:44.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' image = image ']' 2022-04-10T01:33:44.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_32 snapshot 2022-04-10T01:33:44.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:33:44.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:33:44.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_32 2022-04-10T01:33:44.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:33:44.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:33:44.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:33:44.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:33:44.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:33:44.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_32 snapshot 2022-04-10T01:33:45.387 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:33:45.395 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_32 2022-04-10T01:33:45.432 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_32': 2022-04-10T01:33:45.432 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:33:45.433 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:33:45.433 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:33:45.433 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 138354c62216 2022-04-10T01:33:45.434 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.138354c62216 2022-04-10T01:33:45.434 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:33:45.435 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:33:45.435 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:33:45.436 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:33:45.436 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:33:44 2022 2022-04-10T01:33:45.436 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:33:44 2022 2022-04-10T01:33:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:33:44 2022 2022-04-10T01:33:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:33:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:33:45.438 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 34f68cc3-808d-4b7f-a2e3-415ae64356a9 2022-04-10T01:33:45.438 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:33:45.440 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:33:45.440 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_32 2022-04-10T01:33:45.441 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:33:45.441 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:33:45.441 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_32 2022-04-10T01:33:45.441 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:33:45.442 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=14 2022-04-10T01:33:45.442 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:33:45.442 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:33:45.442 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:33:45.443 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:33:45.443 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:33:45.443 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_32 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:33:45.610 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 30495744 bytes 2022-04-10T01:33:45.786 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 84510720 bytes 2022-04-10T01:33:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 117521408 bytes 2022-04-10T01:33:46.113 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 154283008 bytes 2022-04-10T01:33:46.303 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 223563776 bytes 2022-04-10T01:33:46.451 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 260027392 bytes 2022-04-10T01:33:46.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 264473600 bytes 2022-04-10T01:33:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 279223296 bytes 2022-04-10T01:33:46.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 317742080 bytes 2022-04-10T01:33:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 365018112 bytes 2022-04-10T01:33:47.185 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 428400640 bytes 2022-04-10T01:33:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 475247616 bytes 2022-04-10T01:33:47.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 526896128 bytes 2022-04-10T01:33:47.752 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 626287616 bytes 2022-04-10T01:33:47.862 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 637126656 bytes 2022-04-10T01:33:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 669275136 bytes 2022-04-10T01:33:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 722989056 bytes 2022-04-10T01:33:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 760867840 bytes 2022-04-10T01:33:48.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 815357952 bytes 2022-04-10T01:33:48.752 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 919194624 bytes 2022-04-10T01:33:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 960067584 bytes 2022-04-10T01:33:49.127 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 1044002816 bytes 2022-04-10T01:33:49.289 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 1085671424 bytes 2022-04-10T01:33:49.465 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 1143140352 bytes 2022-04-10T01:33:49.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 1255730176 bytes 2022-04-10T01:33:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1308950528 bytes 2022-04-10T01:33:49.994 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1309594624 bytes 2022-04-10T01:33:50.126 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1331715072 bytes 2022-04-10T01:33:50.279 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1364077568 bytes 2022-04-10T01:33:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1417501696 bytes 2022-04-10T01:33:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1486978048 bytes 2022-04-10T01:33:50.833 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1543805952 bytes 2022-04-10T01:33:50.995 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1583648768 bytes 2022-04-10T01:33:51.173 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1647038464 bytes 2022-04-10T01:33:51.298 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1659578368 bytes 2022-04-10T01:33:51.416 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1672425472 bytes 2022-04-10T01:33:51.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1743233024 bytes 2022-04-10T01:33:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1759961088 bytes 2022-04-10T01:33:51.921 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1835825152 bytes 2022-04-10T01:33:52.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1955042304 bytes 2022-04-10T01:33:52.331 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1974921216 bytes 2022-04-10T01:33:52.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 2071281664 bytes 2022-04-10T01:33:52.707 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 2106311680 bytes 2022-04-10T01:33:52.947 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 2213574656 bytes 2022-04-10T01:33:53.096 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 2243196928 bytes 2022-04-10T01:33:53.305 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 2326048768 bytes 2022-04-10T01:33:53.442 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 2348644352 bytes 2022-04-10T01:33:53.603 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 2406425600 bytes 2022-04-10T01:33:53.782 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 2445347840 bytes 2022-04-10T01:33:53.921 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 2474711040 bytes 2022-04-10T01:33:54.047 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 2491004928 bytes 2022-04-10T01:33:54.193 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 2522532864 bytes 2022-04-10T01:33:54.389 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 2599840768 bytes 2022-04-10T01:33:54.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5400 IOs, 2633785344 bytes 2022-04-10T01:33:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5500 IOs, 2665449472 bytes 2022-04-10T01:33:54.850 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5600 IOs, 2719225856 bytes 2022-04-10T01:33:55.016 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5700 IOs, 2765024256 bytes 2022-04-10T01:33:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5800 IOs, 2790226944 bytes 2022-04-10T01:33:55.299 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5900 IOs, 2813335552 bytes 2022-04-10T01:33:55.439 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6000 IOs, 2834053120 bytes 2022-04-10T01:33:55.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6100 IOs, 2910391296 bytes 2022-04-10T01:33:55.753 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6200 IOs, 2948105216 bytes 2022-04-10T01:33:55.949 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6300 IOs, 2994942976 bytes 2022-04-10T01:33:56.121 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6400 IOs, 3046498304 bytes 2022-04-10T01:33:56.232 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6500 IOs, 3047097344 bytes 2022-04-10T01:33:56.365 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6600 IOs, 3066710016 bytes 2022-04-10T01:33:56.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6700 IOs, 3146850304 bytes 2022-04-10T01:33:56.770 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6800 IOs, 3236512768 bytes 2022-04-10T01:33:56.963 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6900 IOs, 3296023552 bytes 2022-04-10T01:33:57.104 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7000 IOs, 3321911296 bytes 2022-04-10T01:33:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7100 IOs, 3364535296 bytes 2022-04-10T01:33:57.431 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7200 IOs, 3413990400 bytes 2022-04-10T01:33:57.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7300 IOs, 3482073088 bytes 2022-04-10T01:33:57.832 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7400 IOs, 3555405824 bytes 2022-04-10T01:33:57.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7500 IOs, 3559222272 bytes 2022-04-10T01:33:58.145 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7600 IOs, 3630320640 bytes 2022-04-10T01:33:58.270 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7700 IOs, 3640368128 bytes 2022-04-10T01:33:58.389 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7800 IOs, 3651281920 bytes 2022-04-10T01:33:58.528 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7900 IOs, 3677862912 bytes 2022-04-10T01:33:58.722 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8000 IOs, 3763794944 bytes 2022-04-10T01:33:58.905 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8100 IOs, 3813671936 bytes 2022-04-10T01:33:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8200 IOs, 3817794560 bytes 2022-04-10T01:33:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8300 IOs, 3882518528 bytes 2022-04-10T01:33:59.331 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8400 IOs, 3897379840 bytes 2022-04-10T01:33:59.444 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:33:59.445 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:33:59.445 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:33:59.445 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:33:59.446 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:33:59.446 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_33 2022-04-10T01:33:59.446 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image_and_enable_mirror cluster2 mirror image_33 snapshot 128M 2022-04-10T01:33:59.446 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:33:59.446 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:33:59.447 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:33:59.447 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:33:59.447 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_33 2022-04-10T01:33:59.447 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:33:59.448 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:33:59.448 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:33:59.448 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:33:59.450 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:33:59.450 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:33:59.451 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snapshot ']' 2022-04-10T01:33:59.451 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:33:59.451 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_33 128M 2022-04-10T01:33:59.451 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:33:59.452 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:33:59.452 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:33:59.452 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:33:59.452 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_33 2022-04-10T01:33:59.453 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:33:59.453 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:33:59.453 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:33:59.453 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:33:59.453 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:33:59.454 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:33:59.454 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 128M ']' 2022-04-10T01:33:59.454 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=128M 2022-04-10T01:33:59.454 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:33:59.455 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_33 2022-04-10T01:33:59.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' image = image ']' 2022-04-10T01:33:59.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_33 snapshot 2022-04-10T01:33:59.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:33:59.489 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:33:59.489 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_33 2022-04-10T01:33:59.489 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:33:59.490 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:33:59.490 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:33:59.490 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:33:59.490 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:33:59.491 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_33 snapshot 2022-04-10T01:34:00.427 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:34:00.437 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_33 2022-04-10T01:34:00.472 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_33': 2022-04-10T01:34:00.472 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:34:00.472 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:34:00.473 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:34:00.473 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 138cf06c3ecd 2022-04-10T01:34:00.473 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.138cf06c3ecd 2022-04-10T01:34:00.473 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:34:00.474 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:34:00.474 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:34:00.474 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:34:00.474 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:33:59 2022 2022-04-10T01:34:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:33:59 2022 2022-04-10T01:34:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:33:59 2022 2022-04-10T01:34:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:34:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:34:00.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 3334b7aa-6174-45c6-b4b8-608b7f09db2a 2022-04-10T01:34:00.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:34:00.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:34:00.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_33 2022-04-10T01:34:00.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:34:00.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:34:00.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_33 2022-04-10T01:34:00.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:34:00.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=14 2022-04-10T01:34:00.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:34:00.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:34:00.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:34:00.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:34:00.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:34:00.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_33 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:34:00.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 92011520 bytes 2022-04-10T01:34:00.851 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 110969856 bytes 2022-04-10T01:34:01.051 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 192818176 bytes 2022-04-10T01:34:01.163 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 193445888 bytes 2022-04-10T01:34:01.273 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 197633024 bytes 2022-04-10T01:34:01.396 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 210950144 bytes 2022-04-10T01:34:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 253356032 bytes 2022-04-10T01:34:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 281412608 bytes 2022-04-10T01:34:01.828 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 310256640 bytes 2022-04-10T01:34:01.946 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 321887232 bytes 2022-04-10T01:34:02.161 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 398358528 bytes 2022-04-10T01:34:02.302 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 444110848 bytes 2022-04-10T01:34:02.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 507084800 bytes 2022-04-10T01:34:02.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 551700480 bytes 2022-04-10T01:34:02.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 569156608 bytes 2022-04-10T01:34:02.948 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 600761344 bytes 2022-04-10T01:34:03.117 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 652188672 bytes 2022-04-10T01:34:03.252 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 674669568 bytes 2022-04-10T01:34:03.371 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 686889984 bytes 2022-04-10T01:34:03.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 725170176 bytes 2022-04-10T01:34:03.630 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 725753856 bytes 2022-04-10T01:34:03.792 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 778225664 bytes 2022-04-10T01:34:03.994 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 869635072 bytes 2022-04-10T01:34:04.171 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 909724672 bytes 2022-04-10T01:34:04.325 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 952368128 bytes 2022-04-10T01:34:04.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 989196288 bytes 2022-04-10T01:34:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 998111232 bytes 2022-04-10T01:34:04.716 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1012196352 bytes 2022-04-10T01:34:04.847 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1033083904 bytes 2022-04-10T01:34:04.988 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1060657152 bytes 2022-04-10T01:34:05.113 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1076843520 bytes 2022-04-10T01:34:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1128187904 bytes 2022-04-10T01:34:05.429 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1158919168 bytes 2022-04-10T01:34:05.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1231629312 bytes 2022-04-10T01:34:05.777 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1273803776 bytes 2022-04-10T01:34:05.968 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1346262016 bytes 2022-04-10T01:34:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1358161920 bytes 2022-04-10T01:34:06.270 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1418757120 bytes 2022-04-10T01:34:06.415 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1451744256 bytes 2022-04-10T01:34:06.566 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1489107968 bytes 2022-04-10T01:34:06.706 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1523491840 bytes 2022-04-10T01:34:06.852 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 1554433024 bytes 2022-04-10T01:34:06.987 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 1583958016 bytes 2022-04-10T01:34:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 1584573440 bytes 2022-04-10T01:34:07.300 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 1665009664 bytes 2022-04-10T01:34:07.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 1716005888 bytes 2022-04-10T01:34:07.606 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 1716618240 bytes 2022-04-10T01:34:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 1747446784 bytes 2022-04-10T01:34:07.922 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 1796299776 bytes 2022-04-10T01:34:08.027 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 1800954880 bytes 2022-04-10T01:34:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 1870915584 bytes 2022-04-10T01:34:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 1922241536 bytes 2022-04-10T01:34:08.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 1969916928 bytes 2022-04-10T01:34:08.728 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5400 IOs, 2009196544 bytes 2022-04-10T01:34:08.948 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5500 IOs, 2096180224 bytes 2022-04-10T01:34:09.129 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5600 IOs, 2158622720 bytes 2022-04-10T01:34:09.291 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5700 IOs, 2205179904 bytes 2022-04-10T01:34:09.434 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5800 IOs, 2231292928 bytes 2022-04-10T01:34:09.615 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5900 IOs, 2298266624 bytes 2022-04-10T01:34:09.810 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6000 IOs, 2390427648 bytes 2022-04-10T01:34:09.986 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6100 IOs, 2429070336 bytes 2022-04-10T01:34:10.179 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6200 IOs, 2495609856 bytes 2022-04-10T01:34:10.368 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6300 IOs, 2552855552 bytes 2022-04-10T01:34:10.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6400 IOs, 2582551552 bytes 2022-04-10T01:34:10.659 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6500 IOs, 2619827200 bytes 2022-04-10T01:34:10.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6600 IOs, 2691990528 bytes 2022-04-10T01:34:11.022 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6700 IOs, 2724941824 bytes 2022-04-10T01:34:11.164 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6800 IOs, 2741831680 bytes 2022-04-10T01:34:11.269 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6900 IOs, 2742451200 bytes 2022-04-10T01:34:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7000 IOs, 2865179648 bytes 2022-04-10T01:34:11.699 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7100 IOs, 2904728576 bytes 2022-04-10T01:34:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7200 IOs, 2933083136 bytes 2022-04-10T01:34:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7300 IOs, 3023669248 bytes 2022-04-10T01:34:12.283 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7400 IOs, 3093335040 bytes 2022-04-10T01:34:12.467 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7500 IOs, 3152470016 bytes 2022-04-10T01:34:12.647 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7600 IOs, 3190821888 bytes 2022-04-10T01:34:12.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7700 IOs, 3246766080 bytes 2022-04-10T01:34:12.944 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7800 IOs, 3278226432 bytes 2022-04-10T01:34:13.083 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7900 IOs, 3300109312 bytes 2022-04-10T01:34:13.202 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8000 IOs, 3313357824 bytes 2022-04-10T01:34:13.385 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8100 IOs, 3382959104 bytes 2022-04-10T01:34:13.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8200 IOs, 3425292288 bytes 2022-04-10T01:34:13.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8300 IOs, 3479323648 bytes 2022-04-10T01:34:13.862 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8400 IOs, 3521718272 bytes 2022-04-10T01:34:14.063 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8500 IOs, 3607539712 bytes 2022-04-10T01:34:14.285 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8600 IOs, 3695638528 bytes 2022-04-10T01:34:14.415 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8700 IOs, 3716113408 bytes 2022-04-10T01:34:14.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:34:14.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:34:14.486 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:34:14.486 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:34:14.486 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:34:14.486 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_34 2022-04-10T01:34:14.487 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image_and_enable_mirror cluster2 mirror image_34 snapshot 128M 2022-04-10T01:34:14.487 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:34:14.487 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:34:14.487 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:34:14.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:34:14.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_34 2022-04-10T01:34:14.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:34:14.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:34:14.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:34:14.489 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:34:14.489 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:34:14.489 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:34:14.489 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snapshot ']' 2022-04-10T01:34:14.490 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:34:14.490 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_34 128M 2022-04-10T01:34:14.490 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:34:14.490 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:34:14.490 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:34:14.491 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:34:14.491 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_34 2022-04-10T01:34:14.491 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:34:14.491 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:34:14.492 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:34:14.492 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:34:14.492 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:34:14.492 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:34:14.492 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 128M ']' 2022-04-10T01:34:14.493 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=128M 2022-04-10T01:34:14.493 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:34:14.493 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_34 2022-04-10T01:34:14.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' image = image ']' 2022-04-10T01:34:14.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_34 snapshot 2022-04-10T01:34:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:34:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:34:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_34 2022-04-10T01:34:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:34:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:34:14.529 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:34:14.529 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:34:14.529 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:34:14.529 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_34 snapshot 2022-04-10T01:34:15.402 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:34:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_34 2022-04-10T01:34:15.448 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_34': 2022-04-10T01:34:15.448 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:34:15.449 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:34:15.449 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:34:15.449 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 1397675f08d2 2022-04-10T01:34:15.449 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.1397675f08d2 2022-04-10T01:34:15.449 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:34:15.450 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:34:15.450 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:34:15.450 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:34:15.451 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:34:14 2022 2022-04-10T01:34:15.451 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:34:14 2022 2022-04-10T01:34:15.451 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:34:14 2022 2022-04-10T01:34:15.451 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:34:15.452 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:34:15.452 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: fc8271db-fe00-47f6-a840-9d97ed18fbb7 2022-04-10T01:34:15.452 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:34:15.454 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:34:15.454 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_34 2022-04-10T01:34:15.455 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:34:15.455 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:34:15.455 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_34 2022-04-10T01:34:15.456 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:34:15.456 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=12 2022-04-10T01:34:15.456 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:34:15.456 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:34:15.457 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:34:15.457 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:34:15.457 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:34:15.457 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_34 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:34:15.614 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 14845952 bytes 2022-04-10T01:34:15.752 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 51182592 bytes 2022-04-10T01:34:15.923 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 108984320 bytes 2022-04-10T01:34:16.079 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 153737216 bytes 2022-04-10T01:34:16.233 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 193659904 bytes 2022-04-10T01:34:16.343 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 196770816 bytes 2022-04-10T01:34:16.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 260458496 bytes 2022-04-10T01:34:16.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 291269632 bytes 2022-04-10T01:34:16.848 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 356205568 bytes 2022-04-10T01:34:16.980 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 389383168 bytes 2022-04-10T01:34:17.124 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 424390656 bytes 2022-04-10T01:34:17.298 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 478285824 bytes 2022-04-10T01:34:17.406 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 483256320 bytes 2022-04-10T01:34:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 493976576 bytes 2022-04-10T01:34:17.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 508936192 bytes 2022-04-10T01:34:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 580967424 bytes 2022-04-10T01:34:18.051 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 662258688 bytes 2022-04-10T01:34:18.173 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 679031808 bytes 2022-04-10T01:34:18.363 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 767466496 bytes 2022-04-10T01:34:18.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 812115968 bytes 2022-04-10T01:34:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 868325376 bytes 2022-04-10T01:34:18.855 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 922044416 bytes 2022-04-10T01:34:18.993 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 945783808 bytes 2022-04-10T01:34:19.117 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 963461120 bytes 2022-04-10T01:34:19.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 971523072 bytes 2022-04-10T01:34:19.411 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1026510848 bytes 2022-04-10T01:34:19.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1070311424 bytes 2022-04-10T01:34:19.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1071636480 bytes 2022-04-10T01:34:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1133158400 bytes 2022-04-10T01:34:20.043 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1191464960 bytes 2022-04-10T01:34:20.202 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1248620544 bytes 2022-04-10T01:34:20.391 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1301202944 bytes 2022-04-10T01:34:20.541 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1332497408 bytes 2022-04-10T01:34:20.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1384995840 bytes 2022-04-10T01:34:20.838 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1401317376 bytes 2022-04-10T01:34:20.973 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1439470592 bytes 2022-04-10T01:34:21.142 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1475245056 bytes 2022-04-10T01:34:21.279 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1501235200 bytes 2022-04-10T01:34:21.411 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1519327232 bytes 2022-04-10T01:34:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1541559296 bytes 2022-04-10T01:34:21.713 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1601055744 bytes 2022-04-10T01:34:21.889 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 1646605312 bytes 2022-04-10T01:34:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 1721285632 bytes 2022-04-10T01:34:22.227 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 1750117376 bytes 2022-04-10T01:34:22.382 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 1795544064 bytes 2022-04-10T01:34:22.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 1843420160 bytes 2022-04-10T01:34:22.719 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 1901827072 bytes 2022-04-10T01:34:22.845 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 1912170496 bytes 2022-04-10T01:34:22.990 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 1945858048 bytes 2022-04-10T01:34:23.199 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 2033180672 bytes 2022-04-10T01:34:23.347 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 2071113728 bytes 2022-04-10T01:34:23.428 INFO:tasks.ceph.cluster2.osd.2.smithi043.stderr:2022-04-10T01:34:23.425+0000 7f8be0453700 -1 reset not still connected to 0x55f2a45bef70 2022-04-10T01:34:23.461 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 2076266496 bytes 2022-04-10T01:34:23.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 2131786752 bytes 2022-04-10T01:34:23.789 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5400 IOs, 2165301248 bytes 2022-04-10T01:34:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5500 IOs, 2208247808 bytes 2022-04-10T01:34:24.065 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5600 IOs, 2217896960 bytes 2022-04-10T01:34:24.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5700 IOs, 2230367232 bytes 2022-04-10T01:34:24.334 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5800 IOs, 2256256000 bytes 2022-04-10T01:34:24.489 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5900 IOs, 2299372544 bytes 2022-04-10T01:34:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6000 IOs, 2356837376 bytes 2022-04-10T01:34:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6100 IOs, 2357462016 bytes 2022-04-10T01:34:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6200 IOs, 2413159424 bytes 2022-04-10T01:34:25.068 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6300 IOs, 2425356288 bytes 2022-04-10T01:34:25.194 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6400 IOs, 2442599424 bytes 2022-04-10T01:34:25.349 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6500 IOs, 2497641472 bytes 2022-04-10T01:34:25.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6600 IOs, 2498260992 bytes 2022-04-10T01:34:25.628 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6700 IOs, 2531280896 bytes 2022-04-10T01:34:25.786 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6800 IOs, 2569645056 bytes 2022-04-10T01:34:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6900 IOs, 2595191808 bytes 2022-04-10T01:34:26.034 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7000 IOs, 2595771392 bytes 2022-04-10T01:34:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7100 IOs, 2724020224 bytes 2022-04-10T01:34:26.414 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7200 IOs, 2735403008 bytes 2022-04-10T01:34:26.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7300 IOs, 2790060032 bytes 2022-04-10T01:34:26.726 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7400 IOs, 2815967232 bytes 2022-04-10T01:34:26.861 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7500 IOs, 2840668160 bytes 2022-04-10T01:34:27.041 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7600 IOs, 2908318720 bytes 2022-04-10T01:34:27.150 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7700 IOs, 2908937216 bytes 2022-04-10T01:34:27.312 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7800 IOs, 2955084800 bytes 2022-04-10T01:34:27.461 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:34:27.461 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:34:27.461 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:34:27.462 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:34:27.462 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:34:27.462 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_35 2022-04-10T01:34:27.462 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image_and_enable_mirror cluster2 mirror image_35 snapshot 128M 2022-04-10T01:34:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:34:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:34:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:34:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:34:27.464 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_35 2022-04-10T01:34:27.464 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:34:27.464 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:34:27.464 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:34:27.465 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:34:27.465 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:34:27.465 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:34:27.465 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snapshot ']' 2022-04-10T01:34:27.466 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:34:27.466 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_35 128M 2022-04-10T01:34:27.467 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:34:27.467 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:34:27.467 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:34:27.467 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:34:27.468 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_35 2022-04-10T01:34:27.468 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:34:27.468 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:34:27.468 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:34:27.468 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:34:27.469 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:34:27.469 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:34:27.469 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 128M ']' 2022-04-10T01:34:27.469 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=128M 2022-04-10T01:34:27.470 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:34:27.470 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_35 2022-04-10T01:34:27.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' image = image ']' 2022-04-10T01:34:27.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_35 snapshot 2022-04-10T01:34:27.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:34:27.505 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:34:27.505 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_35 2022-04-10T01:34:27.505 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:34:27.505 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:34:27.506 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:34:27.506 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:34:27.506 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:34:27.506 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_35 snapshot 2022-04-10T01:34:28.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:34:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_35 2022-04-10T01:34:28.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_35': 2022-04-10T01:34:28.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:34:28.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:34:28.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:34:28.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 139f5c674309 2022-04-10T01:34:28.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.139f5c674309 2022-04-10T01:34:28.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:34:28.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:34:28.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:34:28.566 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:34:28.566 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:34:27 2022 2022-04-10T01:34:28.566 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:34:27 2022 2022-04-10T01:34:28.566 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:34:27 2022 2022-04-10T01:34:28.567 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:34:28.567 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:34:28.567 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 2864f9d0-8af5-4721-a606-e52b47ecb56e 2022-04-10T01:34:28.567 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:34:28.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:34:28.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_35 2022-04-10T01:34:28.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:34:28.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:34:28.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_35 2022-04-10T01:34:28.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:34:28.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=11 2022-04-10T01:34:28.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:34:28.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:34:28.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:34:28.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:34:28.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:34:28.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_35 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:34:28.763 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 51855360 bytes 2022-04-10T01:34:28.900 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 75652096 bytes 2022-04-10T01:34:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 128249856 bytes 2022-04-10T01:34:29.286 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 261969920 bytes 2022-04-10T01:34:29.464 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 306011136 bytes 2022-04-10T01:34:29.630 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 356667392 bytes 2022-04-10T01:34:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 392655872 bytes 2022-04-10T01:34:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 449268736 bytes 2022-04-10T01:34:30.114 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 504700928 bytes 2022-04-10T01:34:30.266 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 539508736 bytes 2022-04-10T01:34:30.370 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 540132352 bytes 2022-04-10T01:34:30.541 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 594427904 bytes 2022-04-10T01:34:30.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 630747136 bytes 2022-04-10T01:34:30.827 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 660163584 bytes 2022-04-10T01:34:30.833 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8897000926169991) > weight (0.25) 2022-04-10T01:34:30.833 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9868924674344269) > weight (0.25) 2022-04-10T01:34:30.833 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.6493956320965029) > weight (0.25) 2022-04-10T01:34:30.834 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.8605644466400687) > weight (0.25) 2022-04-10T01:34:30.834 INFO:tasks.rbd_mirror_thrash:waiting for 65 secs before thrashing 2022-04-10T01:34:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 700440576 bytes 2022-04-10T01:34:31.091 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 710226944 bytes 2022-04-10T01:34:31.288 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 794365952 bytes 2022-04-10T01:34:31.463 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 851400704 bytes 2022-04-10T01:34:31.602 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 879631360 bytes 2022-04-10T01:34:31.745 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 903986176 bytes 2022-04-10T01:34:31.871 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 925524992 bytes 2022-04-10T01:34:32.067 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 1008410624 bytes 2022-04-10T01:34:32.235 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 1064936448 bytes 2022-04-10T01:34:32.424 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 1134865408 bytes 2022-04-10T01:34:32.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 1183597568 bytes 2022-04-10T01:34:32.709 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1201606656 bytes 2022-04-10T01:34:32.826 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1216029696 bytes 2022-04-10T01:34:32.983 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1261938688 bytes 2022-04-10T01:34:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1321106432 bytes 2022-04-10T01:34:33.329 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1377707008 bytes 2022-04-10T01:34:33.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1417869312 bytes 2022-04-10T01:34:33.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1472223232 bytes 2022-04-10T01:34:33.845 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1541600256 bytes 2022-04-10T01:34:34.028 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1613485056 bytes 2022-04-10T01:34:34.165 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1636032512 bytes 2022-04-10T01:34:34.317 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1685563392 bytes 2022-04-10T01:34:34.483 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1735601152 bytes 2022-04-10T01:34:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1755227136 bytes 2022-04-10T01:34:34.755 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1798273024 bytes 2022-04-10T01:34:34.903 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1841574912 bytes 2022-04-10T01:34:35.054 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1878759424 bytes 2022-04-10T01:34:35.163 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 1885756416 bytes 2022-04-10T01:34:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 1890441216 bytes 2022-04-10T01:34:35.427 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 1930394624 bytes 2022-04-10T01:34:35.616 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 2010338304 bytes 2022-04-10T01:34:35.778 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 2052492288 bytes 2022-04-10T01:34:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 2094262272 bytes 2022-04-10T01:34:36.112 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 2178762752 bytes 2022-04-10T01:34:36.310 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 2241219584 bytes 2022-04-10T01:34:36.474 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 2319160320 bytes 2022-04-10T01:34:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 2421420032 bytes 2022-04-10T01:34:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 2489362432 bytes 2022-04-10T01:34:37.069 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 2522565632 bytes 2022-04-10T01:34:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5400 IOs, 2537922560 bytes 2022-04-10T01:34:37.354 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5500 IOs, 2585199616 bytes 2022-04-10T01:34:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5600 IOs, 2600604672 bytes 2022-04-10T01:34:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5700 IOs, 2669566976 bytes 2022-04-10T01:34:37.831 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5800 IOs, 2735852544 bytes 2022-04-10T01:34:38.042 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5900 IOs, 2815252480 bytes 2022-04-10T01:34:38.149 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6000 IOs, 2832677888 bytes 2022-04-10T01:34:38.348 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6100 IOs, 2884225024 bytes 2022-04-10T01:34:38.562 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6200 IOs, 2977037312 bytes 2022-04-10T01:34:38.799 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6300 IOs, 3078011904 bytes 2022-04-10T01:34:38.951 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6400 IOs, 3113624576 bytes 2022-04-10T01:34:39.128 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6500 IOs, 3173618688 bytes 2022-04-10T01:34:39.269 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6600 IOs, 3205796864 bytes 2022-04-10T01:34:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6700 IOs, 3234778112 bytes 2022-04-10T01:34:39.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6800 IOs, 3265726464 bytes 2022-04-10T01:34:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:34:39.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:34:39.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:34:39.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:34:39.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:34:39.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_36 2022-04-10T01:34:39.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image_and_enable_mirror cluster2 mirror image_36 snapshot 128M 2022-04-10T01:34:39.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:34:39.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:34:39.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:34:39.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:34:39.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_36 2022-04-10T01:34:39.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:34:39.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:34:39.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:34:39.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:34:39.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:34:39.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:34:39.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snapshot ']' 2022-04-10T01:34:39.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:34:39.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_36 128M 2022-04-10T01:34:39.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:34:39.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:34:39.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:34:39.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:34:39.583 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_36 2022-04-10T01:34:39.583 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:34:39.583 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:34:39.583 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:34:39.583 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:34:39.584 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:34:39.584 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:34:39.584 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 128M ']' 2022-04-10T01:34:39.584 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=128M 2022-04-10T01:34:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:34:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_36 2022-04-10T01:34:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' image = image ']' 2022-04-10T01:34:39.619 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_36 snapshot 2022-04-10T01:34:39.619 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:34:39.619 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:34:39.619 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_36 2022-04-10T01:34:39.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:34:39.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:34:39.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:34:39.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:34:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:34:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_36 snapshot 2022-04-10T01:34:40.412 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:34:40.421 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_36 2022-04-10T01:34:40.458 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_36': 2022-04-10T01:34:40.459 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:34:40.459 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:34:40.459 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:34:40.459 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 13a8973e0e89 2022-04-10T01:34:40.460 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.13a8973e0e89 2022-04-10T01:34:40.460 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:34:40.460 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:34:40.460 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:34:40.460 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:34:40.461 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:34:39 2022 2022-04-10T01:34:40.461 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:34:39 2022 2022-04-10T01:34:40.461 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:34:39 2022 2022-04-10T01:34:40.461 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:34:40.462 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:34:40.462 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 347e3669-52d8-4e7d-9693-a1920cf47072 2022-04-10T01:34:40.462 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:34:40.464 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:34:40.464 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_36 2022-04-10T01:34:40.464 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:34:40.464 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:34:40.465 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_36 2022-04-10T01:34:40.465 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:34:40.465 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=6 2022-04-10T01:34:40.465 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:34:40.466 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:34:40.466 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:34:40.466 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:34:40.466 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:34:40.467 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_36 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:34:40.647 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 34077696 bytes 2022-04-10T01:34:40.755 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 37884928 bytes 2022-04-10T01:34:40.992 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 154999808 bytes 2022-04-10T01:34:41.247 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 270893056 bytes 2022-04-10T01:34:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 349679616 bytes 2022-04-10T01:34:41.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 370793472 bytes 2022-04-10T01:34:41.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 419969024 bytes 2022-04-10T01:34:42.017 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 469871616 bytes 2022-04-10T01:34:42.142 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 493414400 bytes 2022-04-10T01:34:42.349 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 559591424 bytes 2022-04-10T01:34:42.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 590945280 bytes 2022-04-10T01:34:42.617 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 603023360 bytes 2022-04-10T01:34:42.783 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 654884864 bytes 2022-04-10T01:34:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 726004736 bytes 2022-04-10T01:34:43.080 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 726598656 bytes 2022-04-10T01:34:43.245 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 782690304 bytes 2022-04-10T01:34:43.449 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 860289024 bytes 2022-04-10T01:34:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 925034496 bytes 2022-04-10T01:34:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 943319040 bytes 2022-04-10T01:34:43.890 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 968288256 bytes 2022-04-10T01:34:44.065 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 1024523264 bytes 2022-04-10T01:34:44.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 1097012224 bytes 2022-04-10T01:34:44.419 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 1147204608 bytes 2022-04-10T01:34:44.529 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 1161307136 bytes 2022-04-10T01:34:44.728 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 1224944640 bytes 2022-04-10T01:34:44.965 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1345074176 bytes 2022-04-10T01:34:45.040 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2022-04-10T01:34:45.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2022-04-10T01:34:45.042 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2022-04-10T01:34:45.042 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2022-04-10T01:34:45.042 INFO:teuthology.orchestra.run:waiting for 600 2022-04-10T01:34:45.043 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr:2022-04-10T01:34:45.039+0000 7f089eb66700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 48038) UID: 1000 2022-04-10T01:34:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1402898432 bytes 2022-04-10T01:34:45.374 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1469540352 bytes 2022-04-10T01:34:45.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr:daemon-helper: command failed with exit status 1 2022-04-10T01:34:45.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1492622336 bytes 2022-04-10T01:34:45.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1546187776 bytes 2022-04-10T01:34:45.828 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1577111552 bytes 2022-04-10T01:34:45.988 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1629348864 bytes 2022-04-10T01:34:46.131 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1661215744 bytes 2022-04-10T01:34:46.271 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1679345664 bytes 2022-04-10T01:34:46.447 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1726864384 bytes 2022-04-10T01:34:46.474 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:34:46.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:34:46.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:34:46.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:34:46.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:34:46.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_37 2022-04-10T01:34:46.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image_and_enable_mirror cluster2 mirror image_37 snapshot 128M 2022-04-10T01:34:46.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:34:46.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:34:46.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:34:46.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:34:46.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_37 2022-04-10T01:34:46.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:34:46.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:34:46.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:34:46.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:34:46.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:34:46.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:34:46.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snapshot ']' 2022-04-10T01:34:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:34:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_37 128M 2022-04-10T01:34:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:34:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:34:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:34:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:34:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_37 2022-04-10T01:34:46.486 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:34:46.486 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:34:46.486 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:34:46.487 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:34:46.487 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:34:46.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:34:46.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 128M ']' 2022-04-10T01:34:46.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=128M 2022-04-10T01:34:46.489 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:34:46.489 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_37 2022-04-10T01:34:46.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' image = image ']' 2022-04-10T01:34:46.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_37 snapshot 2022-04-10T01:34:46.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:34:46.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:34:46.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_37 2022-04-10T01:34:46.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:34:46.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:34:46.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:34:46.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:34:46.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:34:46.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_37 snapshot 2022-04-10T01:34:46.631 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:34:46.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_37 2022-04-10T01:34:46.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_37': 2022-04-10T01:34:46.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:34:46.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:34:46.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:34:46.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 13b0f0397412 2022-04-10T01:34:46.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.13b0f0397412 2022-04-10T01:34:46.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:34:46.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:34:46.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:34:46.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:34:46.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:34:46 2022 2022-04-10T01:34:46.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:34:46 2022 2022-04-10T01:34:46.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:34:46 2022 2022-04-10T01:34:46.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:34:46.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:34:46.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 6e506c1e-5b46-476c-afb6-a70d5f9862e5 2022-04-10T01:34:46.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:34:46.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:34:46.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_37 2022-04-10T01:34:46.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:34:46.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:34:46.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_37 2022-04-10T01:34:46.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:34:46.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=9 2022-04-10T01:34:46.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:34:46.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:34:46.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:34:46.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:34:46.681 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:34:46.681 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_37 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:34:46.851 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 30624768 bytes 2022-04-10T01:34:47.046 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 106738688 bytes 2022-04-10T01:34:47.174 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 131256320 bytes 2022-04-10T01:34:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 171371520 bytes 2022-04-10T01:34:47.467 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 207001600 bytes 2022-04-10T01:34:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 267531264 bytes 2022-04-10T01:34:47.741 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 268123136 bytes 2022-04-10T01:34:47.980 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 390437888 bytes 2022-04-10T01:34:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 466677760 bytes 2022-04-10T01:34:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 498463744 bytes 2022-04-10T01:34:48.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 533088256 bytes 2022-04-10T01:34:48.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 591852544 bytes 2022-04-10T01:34:48.842 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 650466304 bytes 2022-04-10T01:34:49.041 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 736638976 bytes 2022-04-10T01:34:49.177 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 753311744 bytes 2022-04-10T01:34:49.349 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 814136320 bytes 2022-04-10T01:34:49.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 866382848 bytes 2022-04-10T01:34:49.648 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 891566080 bytes 2022-04-10T01:34:49.851 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 976411648 bytes 2022-04-10T01:34:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 1007084544 bytes 2022-04-10T01:34:50.103 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 1013647360 bytes 2022-04-10T01:34:50.225 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 1045526528 bytes 2022-04-10T01:34:50.392 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 1080468480 bytes 2022-04-10T01:34:50.532 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 1111387136 bytes 2022-04-10T01:34:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 1188536320 bytes 2022-04-10T01:34:50.868 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1220428800 bytes 2022-04-10T01:34:51.049 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1282421760 bytes 2022-04-10T01:34:51.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-04-10T01:34:51.145 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2022-04-10T01:34:51.145 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2022-04-10T01:34:51.145 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2022-04-10T01:34:51.150 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2022-04-10T01:34:51.150 INFO:tasks.rbd_mirror_thrash:waiting for 98 secs before thrashing 2022-04-10T01:34:51.248 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1361173504 bytes 2022-04-10T01:34:51.411 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1401890816 bytes 2022-04-10T01:34:51.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1454173184 bytes 2022-04-10T01:34:51.715 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1462597632 bytes 2022-04-10T01:34:51.874 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1505993728 bytes 2022-04-10T01:34:52.007 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1531084800 bytes 2022-04-10T01:34:52.133 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1548448768 bytes 2022-04-10T01:34:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1637792768 bytes 2022-04-10T01:34:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1676335104 bytes 2022-04-10T01:34:52.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1726735360 bytes 2022-04-10T01:34:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1760824320 bytes 2022-04-10T01:34:52.950 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1816495104 bytes 2022-04-10T01:34:53.152 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1875825664 bytes 2022-04-10T01:34:53.318 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1928121344 bytes 2022-04-10T01:34:53.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 1974945792 bytes 2022-04-10T01:34:53.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 2038784000 bytes 2022-04-10T01:34:53.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 2083148800 bytes 2022-04-10T01:34:54.002 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 2144034816 bytes 2022-04-10T01:34:54.110 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 2144624640 bytes 2022-04-10T01:34:54.259 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 2187136000 bytes 2022-04-10T01:34:54.436 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 2244749312 bytes 2022-04-10T01:34:54.615 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 2299295744 bytes 2022-04-10T01:34:54.765 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 2336040960 bytes 2022-04-10T01:34:54.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 2407161856 bytes 2022-04-10T01:34:55.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 2514150400 bytes 2022-04-10T01:34:55.303 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 2521972736 bytes 2022-04-10T01:34:55.461 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5400 IOs, 2588092416 bytes 2022-04-10T01:34:55.616 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5500 IOs, 2604900352 bytes 2022-04-10T01:34:55.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:34:55.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:34:55.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:34:55.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:34:55.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:34:55.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_38 2022-04-10T01:34:55.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image_and_enable_mirror cluster2 mirror image_38 snapshot 128M 2022-04-10T01:34:55.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:34:55.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:34:55.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:34:55.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:34:55.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_38 2022-04-10T01:34:55.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:34:55.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:34:55.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:34:55.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:34:55.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:34:55.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:34:55.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snapshot ']' 2022-04-10T01:34:55.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:34:55.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_38 128M 2022-04-10T01:34:55.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:34:55.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:34:55.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:34:55.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:34:55.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_38 2022-04-10T01:34:55.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:34:55.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:34:55.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:34:55.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:34:55.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:34:55.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:34:55.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 128M ']' 2022-04-10T01:34:55.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=128M 2022-04-10T01:34:55.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:34:55.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_38 2022-04-10T01:34:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' image = image ']' 2022-04-10T01:34:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_38 snapshot 2022-04-10T01:34:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:34:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:34:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_38 2022-04-10T01:34:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:34:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:34:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:34:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:34:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:34:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_38 snapshot 2022-04-10T01:34:56.417 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:34:56.425 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_38 2022-04-10T01:34:56.460 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_38': 2022-04-10T01:34:56.460 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:34:56.460 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:34:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:34:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 13c41735f2f5 2022-04-10T01:34:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.13c41735f2f5 2022-04-10T01:34:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:34:56.462 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:34:56.462 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:34:56.462 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:34:56.462 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:34:55 2022 2022-04-10T01:34:56.462 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:34:55 2022 2022-04-10T01:34:56.463 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:34:55 2022 2022-04-10T01:34:56.463 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:34:56.463 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:34:56.463 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 643d8009-5a9c-482d-a7cf-86349b296923 2022-04-10T01:34:56.464 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:34:56.465 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:34:56.466 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_38 2022-04-10T01:34:56.466 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:34:56.466 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:34:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_38 2022-04-10T01:34:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:34:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=6 2022-04-10T01:34:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:34:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:34:56.468 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:34:56.468 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:34:56.468 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:34:56.469 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_38 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:34:56.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 88056832 bytes 2022-04-10T01:34:56.881 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 140791808 bytes 2022-04-10T01:34:57.004 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 179078144 bytes 2022-04-10T01:34:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 229587968 bytes 2022-04-10T01:34:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 316235776 bytes 2022-04-10T01:34:57.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 392134656 bytes 2022-04-10T01:34:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 464929792 bytes 2022-04-10T01:34:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 508271616 bytes 2022-04-10T01:34:58.085 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 543434752 bytes 2022-04-10T01:34:58.269 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 605219840 bytes 2022-04-10T01:34:58.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 611350528 bytes 2022-04-10T01:34:58.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 616182784 bytes 2022-04-10T01:34:58.880 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 687246336 bytes 2022-04-10T01:34:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 736953344 bytes 2022-04-10T01:34:59.153 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 737569792 bytes 2022-04-10T01:34:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 791453696 bytes 2022-04-10T01:34:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 796609536 bytes 2022-04-10T01:34:59.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 819677184 bytes 2022-04-10T01:34:59.706 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 859523072 bytes 2022-04-10T01:34:59.831 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 876771328 bytes 2022-04-10T01:34:59.950 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 888197120 bytes 2022-04-10T01:35:00.084 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 908547072 bytes 2022-04-10T01:35:00.248 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 972481536 bytes 2022-04-10T01:35:00.404 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 997014528 bytes 2022-04-10T01:35:00.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 1047532544 bytes 2022-04-10T01:35:00.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1122874368 bytes 2022-04-10T01:35:00.870 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1125880832 bytes 2022-04-10T01:35:01.059 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1190204416 bytes 2022-04-10T01:35:01.193 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1215337472 bytes 2022-04-10T01:35:01.245 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr:2022-04-10T01:35:01.242+0000 7f52fc8d6700 -1 reset not still connected to 0x5600f479cdd0 2022-04-10T01:35:01.248 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr:2022-04-10T01:35:01.246+0000 7f52fc8d6700 -1 reset not still connected to 0x5600f54a80d0 2022-04-10T01:35:01.249 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr:2022-04-10T01:35:01.246+0000 7f52fc8d6700 -1 reset not still connected to 0x5600f6706dd0 2022-04-10T01:35:01.249 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr:2022-04-10T01:35:01.246+0000 7f52fc8d6700 -1 reset not still connected to 0x5600f6f5fc70 2022-04-10T01:35:01.382 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1285892096 bytes 2022-04-10T01:35:01.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1330347008 bytes 2022-04-10T01:35:01.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1363348480 bytes 2022-04-10T01:35:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1391263744 bytes 2022-04-10T01:35:02.053 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1492142080 bytes 2022-04-10T01:35:02.183 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1522869248 bytes 2022-04-10T01:35:02.385 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1598924800 bytes 2022-04-10T01:35:02.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:35:02.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:35:02.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:35:02.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:35:02.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:35:02.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_39 2022-04-10T01:35:02.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image_and_enable_mirror cluster2 mirror image_39 snapshot 128M 2022-04-10T01:35:02.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:35:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:35:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:35:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:35:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_39 2022-04-10T01:35:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:35:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:35:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:35:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:35:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:35:02.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:35:02.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snapshot ']' 2022-04-10T01:35:02.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:35:02.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_39 128M 2022-04-10T01:35:02.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:35:02.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:35:02.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:35:02.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:35:02.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_39 2022-04-10T01:35:02.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:35:02.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:35:02.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:35:02.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:35:02.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:35:02.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:35:02.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 128M ']' 2022-04-10T01:35:02.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=128M 2022-04-10T01:35:02.483 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:35:02.483 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_39 2022-04-10T01:35:02.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' image = image ']' 2022-04-10T01:35:02.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_39 snapshot 2022-04-10T01:35:02.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:35:02.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:35:02.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_39 2022-04-10T01:35:02.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:35:02.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:35:02.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:35:02.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:35:02.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:35:02.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_39 snapshot 2022-04-10T01:35:03.440 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:35:03.449 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_39 2022-04-10T01:35:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_39': 2022-04-10T01:35:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:35:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:35:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:35:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 13ce47429161 2022-04-10T01:35:03.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.13ce47429161 2022-04-10T01:35:03.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:35:03.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:35:03.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:35:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:35:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:35:02 2022 2022-04-10T01:35:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:35:02 2022 2022-04-10T01:35:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:35:02 2022 2022-04-10T01:35:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:35:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:35:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: e2bbf2e7-46b6-4dd8-b6e1-9dea8cf3b0b4 2022-04-10T01:35:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:35:03.492 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:35:03.492 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_39 2022-04-10T01:35:03.492 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:35:03.493 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:35:03.493 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_39 2022-04-10T01:35:03.493 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:35:03.493 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=10 2022-04-10T01:35:03.493 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:35:03.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:35:03.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:35:03.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:35:03.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:35:03.495 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_39 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:35:03.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 16225280 bytes 2022-04-10T01:35:03.790 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 45880320 bytes 2022-04-10T01:35:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 122393600 bytes 2022-04-10T01:35:04.123 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 153150464 bytes 2022-04-10T01:35:04.323 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 224874496 bytes 2022-04-10T01:35:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 316802048 bytes 2022-04-10T01:35:04.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 330678272 bytes 2022-04-10T01:35:04.788 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 352542720 bytes 2022-04-10T01:35:04.902 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 353147904 bytes 2022-04-10T01:35:05.033 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 375151616 bytes 2022-04-10T01:35:05.218 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 451733504 bytes 2022-04-10T01:35:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 500634624 bytes 2022-04-10T01:35:05.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 583428096 bytes 2022-04-10T01:35:05.754 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 634980352 bytes 2022-04-10T01:35:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 730864640 bytes 2022-04-10T01:35:06.174 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 801540096 bytes 2022-04-10T01:35:06.322 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 833623040 bytes 2022-04-10T01:35:06.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 908121088 bytes 2022-04-10T01:35:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 954381312 bytes 2022-04-10T01:35:06.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 1019745280 bytes 2022-04-10T01:35:07.057 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 1081421824 bytes 2022-04-10T01:35:07.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 1105820672 bytes 2022-04-10T01:35:07.328 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 1120770048 bytes 2022-04-10T01:35:07.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 1178368000 bytes 2022-04-10T01:35:07.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 1178972160 bytes 2022-04-10T01:35:07.752 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1226173440 bytes 2022-04-10T01:35:07.905 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1235438592 bytes 2022-04-10T01:35:08.069 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1285568512 bytes 2022-04-10T01:35:08.199 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1298658304 bytes 2022-04-10T01:35:08.359 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1348959232 bytes 2022-04-10T01:35:08.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1374525440 bytes 2022-04-10T01:35:08.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1413006336 bytes 2022-04-10T01:35:08.911 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1490452480 bytes 2022-04-10T01:35:09.172 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1589390336 bytes 2022-04-10T01:35:09.322 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1623764992 bytes 2022-04-10T01:35:09.466 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1667646464 bytes 2022-04-10T01:35:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1695710208 bytes 2022-04-10T01:35:09.770 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1711214592 bytes 2022-04-10T01:35:09.888 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1717611520 bytes 2022-04-10T01:35:10.000 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1719860224 bytes 2022-04-10T01:35:10.170 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1764072448 bytes 2022-04-10T01:35:10.319 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 1795238912 bytes 2022-04-10T01:35:10.493 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 1846034432 bytes 2022-04-10T01:35:10.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 1873026048 bytes 2022-04-10T01:35:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 1891216384 bytes 2022-04-10T01:35:10.957 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 1954633728 bytes 2022-04-10T01:35:11.168 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 2011433984 bytes 2022-04-10T01:35:11.358 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 2081163264 bytes 2022-04-10T01:35:11.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 2124896256 bytes 2022-04-10T01:35:11.662 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 2143541248 bytes 2022-04-10T01:35:11.923 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 2268837888 bytes 2022-04-10T01:35:12.087 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 2319582208 bytes 2022-04-10T01:35:12.258 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 2368531456 bytes 2022-04-10T01:35:12.434 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5400 IOs, 2418281472 bytes 2022-04-10T01:35:12.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5500 IOs, 2448106496 bytes 2022-04-10T01:35:12.920 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5600 IOs, 2487407616 bytes 2022-04-10T01:35:13.121 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5700 IOs, 2587235328 bytes 2022-04-10T01:35:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5800 IOs, 2638514176 bytes 2022-04-10T01:35:13.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:35:13.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:35:13.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:35:13.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:35:13.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:35:13.500 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_40 2022-04-10T01:35:13.500 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image_and_enable_mirror cluster2 mirror image_40 snapshot 128M 2022-04-10T01:35:13.500 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:35:13.500 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:35:13.501 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:35:13.501 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:35:13.501 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_40 2022-04-10T01:35:13.501 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:35:13.502 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:35:13.502 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:35:13.502 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:35:13.502 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:35:13.502 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:35:13.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snapshot ']' 2022-04-10T01:35:13.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:35:13.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_40 128M 2022-04-10T01:35:13.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:35:13.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:35:13.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:35:13.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:35:13.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_40 2022-04-10T01:35:13.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:35:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:35:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:35:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:35:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:35:13.506 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:35:13.506 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 128M ']' 2022-04-10T01:35:13.506 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=128M 2022-04-10T01:35:13.506 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:35:13.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_40 2022-04-10T01:35:13.541 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' image = image ']' 2022-04-10T01:35:13.541 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_40 snapshot 2022-04-10T01:35:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:35:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:35:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_40 2022-04-10T01:35:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:35:13.543 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:35:13.543 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:35:13.543 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:35:13.543 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:35:13.544 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_40 snapshot 2022-04-10T01:35:14.456 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:35:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_40 2022-04-10T01:35:14.502 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_40': 2022-04-10T01:35:14.502 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:35:14.502 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:35:14.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:35:14.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 13d99e9cdcd2 2022-04-10T01:35:14.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.13d99e9cdcd2 2022-04-10T01:35:14.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:35:14.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:35:14.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:35:14.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:35:14.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:35:13 2022 2022-04-10T01:35:14.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:35:13 2022 2022-04-10T01:35:14.505 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:35:13 2022 2022-04-10T01:35:14.505 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:35:14.505 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:35:14.505 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 778e4963-897c-4526-9fc0-0a90b3039381 2022-04-10T01:35:14.506 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:35:14.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:35:14.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_40 2022-04-10T01:35:14.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:35:14.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:35:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_40 2022-04-10T01:35:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:35:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=6 2022-04-10T01:35:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:35:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:35:14.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:35:14.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:35:14.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:35:14.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_40 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:35:14.698 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 30189568 bytes 2022-04-10T01:35:14.843 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 80764928 bytes 2022-04-10T01:35:14.980 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 106412032 bytes 2022-04-10T01:35:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 166590464 bytes 2022-04-10T01:35:15.302 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 210765824 bytes 2022-04-10T01:35:15.456 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 249860096 bytes 2022-04-10T01:35:15.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 307476480 bytes 2022-04-10T01:35:15.794 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 369952768 bytes 2022-04-10T01:35:15.919 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 394692608 bytes 2022-04-10T01:35:16.057 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 418880512 bytes 2022-04-10T01:35:16.202 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 452401152 bytes 2022-04-10T01:35:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 516200448 bytes 2022-04-10T01:35:16.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 547054592 bytes 2022-04-10T01:35:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 607648768 bytes 2022-04-10T01:35:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 632798208 bytes 2022-04-10T01:35:16.990 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 660260864 bytes 2022-04-10T01:35:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 696916992 bytes 2022-04-10T01:35:17.276 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 716111872 bytes 2022-04-10T01:35:17.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 791263232 bytes 2022-04-10T01:35:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 816930816 bytes 2022-04-10T01:35:17.743 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 836926464 bytes 2022-04-10T01:35:17.962 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 923376640 bytes 2022-04-10T01:35:18.089 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 943102976 bytes 2022-04-10T01:35:18.223 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 968651776 bytes 2022-04-10T01:35:18.395 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 1012043776 bytes 2022-04-10T01:35:18.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1021595648 bytes 2022-04-10T01:35:18.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1052019712 bytes 2022-04-10T01:35:18.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1085138944 bytes 2022-04-10T01:35:19.007 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1183651840 bytes 2022-04-10T01:35:19.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1226913792 bytes 2022-04-10T01:35:19.329 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1253413888 bytes 2022-04-10T01:35:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1299519488 bytes 2022-04-10T01:35:19.621 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1325907968 bytes 2022-04-10T01:35:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1380060160 bytes 2022-04-10T01:35:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1421018112 bytes 2022-04-10T01:35:20.115 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1474467840 bytes 2022-04-10T01:35:20.273 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1521630208 bytes 2022-04-10T01:35:20.430 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1559352320 bytes 2022-04-10T01:35:20.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:35:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:35:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:35:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:35:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:35:20.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_41 2022-04-10T01:35:20.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image_and_enable_mirror cluster2 mirror image_41 snapshot 128M 2022-04-10T01:35:20.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:35:20.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:35:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:35:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:35:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_41 2022-04-10T01:35:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:35:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:35:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:35:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:35:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:35:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:35:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snapshot ']' 2022-04-10T01:35:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:35:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_41 128M 2022-04-10T01:35:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:35:20.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:35:20.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:35:20.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:35:20.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_41 2022-04-10T01:35:20.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:35:20.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:35:20.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:35:20.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:35:20.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:35:20.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:35:20.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 128M ']' 2022-04-10T01:35:20.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=128M 2022-04-10T01:35:20.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:35:20.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_41 2022-04-10T01:35:20.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' image = image ']' 2022-04-10T01:35:20.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_41 snapshot 2022-04-10T01:35:20.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:35:20.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:35:20.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_41 2022-04-10T01:35:20.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:35:20.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:35:20.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:35:20.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:35:20.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:35:20.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_41 snapshot 2022-04-10T01:35:21.469 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:35:21.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_41 2022-04-10T01:35:21.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_41': 2022-04-10T01:35:21.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:35:21.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:35:21.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:35:21.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 13e194f8b2cc 2022-04-10T01:35:21.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.13e194f8b2cc 2022-04-10T01:35:21.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:35:21.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:35:21.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:35:21.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:35:21.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:35:20 2022 2022-04-10T01:35:21.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:35:20 2022 2022-04-10T01:35:21.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:35:20 2022 2022-04-10T01:35:21.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:35:21.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:35:21.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: af549731-3fee-45da-9830-6e51d5343406 2022-04-10T01:35:21.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:35:21.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:35:21.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_41 2022-04-10T01:35:21.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:35:21.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:35:21.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_41 2022-04-10T01:35:21.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:35:21.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=8 2022-04-10T01:35:21.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:35:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:35:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:35:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:35:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:35:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_41 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:35:21.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 69844992 bytes 2022-04-10T01:35:21.906 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 110367744 bytes 2022-04-10T01:35:22.028 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 133993472 bytes 2022-04-10T01:35:22.186 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 170618880 bytes 2022-04-10T01:35:22.322 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 212918272 bytes 2022-04-10T01:35:22.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 301395968 bytes 2022-04-10T01:35:22.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 327885824 bytes 2022-04-10T01:35:22.889 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 411016192 bytes 2022-04-10T01:35:23.033 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 430944256 bytes 2022-04-10T01:35:23.199 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 477250560 bytes 2022-04-10T01:35:23.369 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 525074432 bytes 2022-04-10T01:35:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 569146368 bytes 2022-04-10T01:35:23.704 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 628645888 bytes 2022-04-10T01:35:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 676925440 bytes 2022-04-10T01:35:24.018 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 714103808 bytes 2022-04-10T01:35:24.174 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 756732928 bytes 2022-04-10T01:35:24.294 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 763355136 bytes 2022-04-10T01:35:24.472 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 824204288 bytes 2022-04-10T01:35:24.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 884824064 bytes 2022-04-10T01:35:24.799 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 910294016 bytes 2022-04-10T01:35:25.027 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 1006650368 bytes 2022-04-10T01:35:25.188 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 1053020160 bytes 2022-04-10T01:35:25.329 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 1077287936 bytes 2022-04-10T01:35:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 1125316608 bytes 2022-04-10T01:35:25.677 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 1188632576 bytes 2022-04-10T01:35:25.799 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1199114240 bytes 2022-04-10T01:35:25.903 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1201345536 bytes 2022-04-10T01:35:26.016 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1208252416 bytes 2022-04-10T01:35:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1244016640 bytes 2022-04-10T01:35:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1246874624 bytes 2022-04-10T01:35:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1264643072 bytes 2022-04-10T01:35:26.586 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1327562752 bytes 2022-04-10T01:35:26.726 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1356007424 bytes 2022-04-10T01:35:26.839 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1362274304 bytes 2022-04-10T01:35:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1384659968 bytes 2022-04-10T01:35:27.172 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1458111488 bytes 2022-04-10T01:35:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1573169152 bytes 2022-04-10T01:35:27.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1592852480 bytes 2022-04-10T01:35:27.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1604599808 bytes 2022-04-10T01:35:27.823 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1640171520 bytes 2022-04-10T01:35:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1665361920 bytes 2022-04-10T01:35:28.068 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 1665983488 bytes 2022-04-10T01:35:28.297 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 1768037376 bytes 2022-04-10T01:35:28.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 1844376576 bytes 2022-04-10T01:35:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 1911863296 bytes 2022-04-10T01:35:28.812 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 1931096064 bytes 2022-04-10T01:35:28.932 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 1939659776 bytes 2022-04-10T01:35:29.108 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 1990096896 bytes 2022-04-10T01:35:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 2012307456 bytes 2022-04-10T01:35:29.375 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 2036286464 bytes 2022-04-10T01:35:29.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 2052518912 bytes 2022-04-10T01:35:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:35:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:35:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:35:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:35:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:35:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_42 2022-04-10T01:35:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image_and_enable_mirror cluster2 mirror image_42 snapshot 128M 2022-04-10T01:35:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:35:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:35:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:35:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:35:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_42 2022-04-10T01:35:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:35:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:35:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:35:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:35:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:35:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:35:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snapshot ']' 2022-04-10T01:35:29.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:35:29.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_42 128M 2022-04-10T01:35:29.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:35:29.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:35:29.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:35:29.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:35:29.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_42 2022-04-10T01:35:29.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:35:29.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:35:29.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:35:29.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:35:29.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:35:29.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:35:29.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 128M ']' 2022-04-10T01:35:29.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=128M 2022-04-10T01:35:29.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:35:29.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_42 2022-04-10T01:35:29.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' image = image ']' 2022-04-10T01:35:29.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_42 snapshot 2022-04-10T01:35:29.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:35:29.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:35:29.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_42 2022-04-10T01:35:29.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:35:29.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:35:29.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:35:29.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:35:29.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:35:29.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_42 snapshot 2022-04-10T01:35:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:35:30.495 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_42 2022-04-10T01:35:30.530 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_42': 2022-04-10T01:35:30.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:35:30.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:35:30.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:35:30.532 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 13e9147fb59 2022-04-10T01:35:30.532 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.13e9147fb59 2022-04-10T01:35:30.532 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:35:30.532 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:35:30.532 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:35:30.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:35:30.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:35:29 2022 2022-04-10T01:35:30.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:35:29 2022 2022-04-10T01:35:30.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:35:29 2022 2022-04-10T01:35:30.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:35:30.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:35:30.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: ceed605c-dff2-49f6-bdf7-f3ed1ce8dfcd 2022-04-10T01:35:30.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:35:30.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:35:30.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_42 2022-04-10T01:35:30.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:35:30.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:35:30.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_42 2022-04-10T01:35:30.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:35:30.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=10 2022-04-10T01:35:30.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:35:30.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:35:30.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:35:30.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:35:30.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:35:30.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_42 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:35:30.720 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 31356928 bytes 2022-04-10T01:35:30.862 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 68680704 bytes 2022-04-10T01:35:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 94233600 bytes 2022-04-10T01:35:31.140 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 130056192 bytes 2022-04-10T01:35:31.324 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 196024320 bytes 2022-04-10T01:35:31.447 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 209166336 bytes 2022-04-10T01:35:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 240210944 bytes 2022-04-10T01:35:31.744 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 276820992 bytes 2022-04-10T01:35:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 328720384 bytes 2022-04-10T01:35:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 367110144 bytes 2022-04-10T01:35:32.172 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 367731712 bytes 2022-04-10T01:35:32.321 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 398984192 bytes 2022-04-10T01:35:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 490485760 bytes 2022-04-10T01:35:32.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 507600896 bytes 2022-04-10T01:35:32.832 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 541297664 bytes 2022-04-10T01:35:32.949 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 552202240 bytes 2022-04-10T01:35:33.053 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 552801280 bytes 2022-04-10T01:35:33.223 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 610155520 bytes 2022-04-10T01:35:33.396 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 649648128 bytes 2022-04-10T01:35:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 703084544 bytes 2022-04-10T01:35:33.735 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 766839808 bytes 2022-04-10T01:35:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 825348096 bytes 2022-04-10T01:35:34.031 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 831312896 bytes 2022-04-10T01:35:34.201 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 888571904 bytes 2022-04-10T01:35:34.315 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 894843904 bytes 2022-04-10T01:35:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 961911808 bytes 2022-04-10T01:35:34.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1032580096 bytes 2022-04-10T01:35:34.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1058964480 bytes 2022-04-10T01:35:34.961 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1076102144 bytes 2022-04-10T01:35:35.106 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1131448320 bytes 2022-04-10T01:35:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1151340544 bytes 2022-04-10T01:35:35.415 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1200043008 bytes 2022-04-10T01:35:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1255131136 bytes 2022-04-10T01:35:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1264932864 bytes 2022-04-10T01:35:35.835 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2022-04-10T01:35:35.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2022-04-10T01:35:35.836 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2022-04-10T01:35:35.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi043.stderr:2022-04-10T01:35:35.837+0000 7f04fc30b700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 49157) UID: 1000 2022-04-10T01:35:35.971 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1378052096 bytes 2022-04-10T01:35:36.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi043.stderr:daemon-helper: command failed with exit status 1 2022-04-10T01:35:36.132 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1405504512 bytes 2022-04-10T01:35:36.298 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1459684352 bytes 2022-04-10T01:35:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1505117184 bytes 2022-04-10T01:35:36.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1505704960 bytes 2022-04-10T01:35:36.733 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1549914112 bytes 2022-04-10T01:35:36.840 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1550532608 bytes 2022-04-10T01:35:36.999 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 1597266944 bytes 2022-04-10T01:35:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 1622024192 bytes 2022-04-10T01:35:37.311 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 1677428736 bytes 2022-04-10T01:35:37.430 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 1690076160 bytes 2022-04-10T01:35:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 1789796352 bytes 2022-04-10T01:35:37.778 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 1802806272 bytes 2022-04-10T01:35:37.964 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 1862132736 bytes 2022-04-10T01:35:38.074 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 1886860288 bytes 2022-04-10T01:35:38.285 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 1952166912 bytes 2022-04-10T01:35:38.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 2011619328 bytes 2022-04-10T01:35:38.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 2078945280 bytes 2022-04-10T01:35:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 2090661888 bytes 2022-04-10T01:35:38.974 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5400 IOs, 2147134464 bytes 2022-04-10T01:35:39.127 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5500 IOs, 2171343872 bytes 2022-04-10T01:35:39.283 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5600 IOs, 2214701056 bytes 2022-04-10T01:35:39.403 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5700 IOs, 2223879168 bytes 2022-04-10T01:35:39.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5800 IOs, 2239632384 bytes 2022-04-10T01:35:39.733 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5900 IOs, 2326044672 bytes 2022-04-10T01:35:39.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6000 IOs, 2358024192 bytes 2022-04-10T01:35:40.009 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6100 IOs, 2386357248 bytes 2022-04-10T01:35:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6200 IOs, 2407555072 bytes 2022-04-10T01:35:40.322 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6300 IOs, 2435616768 bytes 2022-04-10T01:35:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6400 IOs, 2466822144 bytes 2022-04-10T01:35:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:35:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:35:40.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:35:40.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:35:40.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:35:40.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_43 2022-04-10T01:35:40.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image_and_enable_mirror cluster2 mirror image_43 snapshot 128M 2022-04-10T01:35:40.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:35:40.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:35:40.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:35:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:35:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_43 2022-04-10T01:35:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:35:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:35:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:35:40.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:35:40.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:35:40.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:35:40.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snapshot ']' 2022-04-10T01:35:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:35:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_43 128M 2022-04-10T01:35:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:35:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:35:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:35:40.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:35:40.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_43 2022-04-10T01:35:40.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:35:40.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:35:40.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:35:40.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:35:40.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:35:40.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:35:40.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 128M ']' 2022-04-10T01:35:40.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=128M 2022-04-10T01:35:40.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:35:40.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_43 2022-04-10T01:35:40.586 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' image = image ']' 2022-04-10T01:35:40.586 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_43 snapshot 2022-04-10T01:35:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:35:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:35:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_43 2022-04-10T01:35:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:35:40.588 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:35:40.588 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:35:40.588 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:35:40.588 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:35:40.588 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_43 snapshot 2022-04-10T01:35:40.837 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2022-04-10T01:35:40.838 INFO:teuthology.orchestra.run:waiting for 600 2022-04-10T01:35:40.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-04-10T01:35:40.839 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2022-04-10T01:35:40.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2022-04-10T01:35:40.839 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2022-04-10T01:35:40.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2022-04-10T01:35:40.842 INFO:tasks.rbd_mirror_thrash:waiting for 75 secs before thrashing 2022-04-10T01:35:41.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:35:41.585 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_43 2022-04-10T01:35:41.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_43': 2022-04-10T01:35:41.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:35:41.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:35:41.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:35:41.621 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 13f4a7851d76 2022-04-10T01:35:41.621 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.13f4a7851d76 2022-04-10T01:35:41.621 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:35:41.621 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:35:41.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:35:41.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:35:41.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:35:40 2022 2022-04-10T01:35:41.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:35:40 2022 2022-04-10T01:35:41.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:35:40 2022 2022-04-10T01:35:41.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:35:41.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:35:41.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: bcd924f1-5832-47a7-92fc-1dbfdf17dc65 2022-04-10T01:35:41.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:35:41.626 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:35:41.626 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_43 2022-04-10T01:35:41.626 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:35:41.627 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:35:41.627 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_43 2022-04-10T01:35:41.627 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:35:41.627 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=14 2022-04-10T01:35:41.627 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:35:41.628 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:35:41.628 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:35:41.628 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:35:41.628 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:35:41.628 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_43 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:35:41.810 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 32158720 bytes 2022-04-10T01:35:41.914 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 35037184 bytes 2022-04-10T01:35:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 76257280 bytes 2022-04-10T01:35:42.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 157531136 bytes 2022-04-10T01:35:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 201492480 bytes 2022-04-10T01:35:42.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 222686208 bytes 2022-04-10T01:35:42.742 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 281640960 bytes 2022-04-10T01:35:42.853 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 282276864 bytes 2022-04-10T01:35:43.015 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 325872640 bytes 2022-04-10T01:35:43.201 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 390477824 bytes 2022-04-10T01:35:43.360 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 418764800 bytes 2022-04-10T01:35:43.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 449061888 bytes 2022-04-10T01:35:43.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 474281984 bytes 2022-04-10T01:35:43.762 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 485848064 bytes 2022-04-10T01:35:43.892 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 504317952 bytes 2022-04-10T01:35:44.052 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 554347520 bytes 2022-04-10T01:35:44.218 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 599404544 bytes 2022-04-10T01:35:44.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 624594944 bytes 2022-04-10T01:35:44.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 686347264 bytes 2022-04-10T01:35:44.717 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 717007872 bytes 2022-04-10T01:35:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 775751680 bytes 2022-04-10T01:35:45.024 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 801150976 bytes 2022-04-10T01:35:45.211 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 863599616 bytes 2022-04-10T01:35:45.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 902856704 bytes 2022-04-10T01:35:45.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 950737920 bytes 2022-04-10T01:35:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 951585792 bytes 2022-04-10T01:35:45.796 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1009936384 bytes 2022-04-10T01:35:45.922 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1010550784 bytes 2022-04-10T01:35:46.127 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1097377792 bytes 2022-04-10T01:35:46.334 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1178237952 bytes 2022-04-10T01:35:46.465 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1202680832 bytes 2022-04-10T01:35:46.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1232372736 bytes 2022-04-10T01:35:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1292720128 bytes 2022-04-10T01:35:46.963 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1365472256 bytes 2022-04-10T01:35:47.101 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1366064128 bytes 2022-04-10T01:35:47.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1387664384 bytes 2022-04-10T01:35:47.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1418798080 bytes 2022-04-10T01:35:47.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1419436032 bytes 2022-04-10T01:35:47.626 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1451031552 bytes 2022-04-10T01:35:47.863 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1571234816 bytes 2022-04-10T01:35:48.036 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1611963392 bytes 2022-04-10T01:35:48.156 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 1623572480 bytes 2022-04-10T01:35:48.289 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 1675107328 bytes 2022-04-10T01:35:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 1691131904 bytes 2022-04-10T01:35:48.625 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 1756060672 bytes 2022-04-10T01:35:48.812 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 1816412160 bytes 2022-04-10T01:35:48.947 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 1848187904 bytes 2022-04-10T01:35:49.120 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 1903470592 bytes 2022-04-10T01:35:49.319 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 1966034944 bytes 2022-04-10T01:35:49.440 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 1981091840 bytes 2022-04-10T01:35:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 2006297600 bytes 2022-04-10T01:35:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 2012707840 bytes 2022-04-10T01:35:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 2108249088 bytes 2022-04-10T01:35:50.068 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5400 IOs, 2144908288 bytes 2022-04-10T01:35:50.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5500 IOs, 2220687360 bytes 2022-04-10T01:35:50.403 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5600 IOs, 2251664384 bytes 2022-04-10T01:35:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5700 IOs, 2318049280 bytes 2022-04-10T01:35:50.742 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5800 IOs, 2359571456 bytes 2022-04-10T01:35:50.896 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5900 IOs, 2399391744 bytes 2022-04-10T01:35:51.024 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6000 IOs, 2417355776 bytes 2022-04-10T01:35:51.170 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6100 IOs, 2450372608 bytes 2022-04-10T01:35:51.359 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6200 IOs, 2517917696 bytes 2022-04-10T01:35:51.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6300 IOs, 2558628864 bytes 2022-04-10T01:35:51.627 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6400 IOs, 2560609280 bytes 2022-04-10T01:35:51.777 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6500 IOs, 2597901312 bytes 2022-04-10T01:35:51.888 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6600 IOs, 2602245120 bytes 2022-04-10T01:35:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6700 IOs, 2639063040 bytes 2022-04-10T01:35:52.171 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6800 IOs, 2662230016 bytes 2022-04-10T01:35:52.341 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6900 IOs, 2707374080 bytes 2022-04-10T01:35:52.449 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7000 IOs, 2707977216 bytes 2022-04-10T01:35:52.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7100 IOs, 2745702400 bytes 2022-04-10T01:35:52.764 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7200 IOs, 2796389376 bytes 2022-04-10T01:35:52.887 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7300 IOs, 2827029504 bytes 2022-04-10T01:35:53.021 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7400 IOs, 2841870336 bytes 2022-04-10T01:35:53.156 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7500 IOs, 2874210304 bytes 2022-04-10T01:35:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7600 IOs, 2878796800 bytes 2022-04-10T01:35:53.435 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7700 IOs, 2914116608 bytes 2022-04-10T01:35:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7800 IOs, 2921725952 bytes 2022-04-10T01:35:53.704 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7900 IOs, 2962226176 bytes 2022-04-10T01:35:53.844 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8000 IOs, 2999057408 bytes 2022-04-10T01:35:53.971 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8100 IOs, 3004313600 bytes 2022-04-10T01:35:54.085 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8200 IOs, 3009464320 bytes 2022-04-10T01:35:54.275 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8300 IOs, 3082023936 bytes 2022-04-10T01:35:54.492 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8400 IOs, 3171316736 bytes 2022-04-10T01:35:54.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8500 IOs, 3210125312 bytes 2022-04-10T01:35:54.769 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8600 IOs, 3225275392 bytes 2022-04-10T01:35:55.005 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8700 IOs, 3344206848 bytes 2022-04-10T01:35:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8800 IOs, 3354383360 bytes 2022-04-10T01:35:55.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8900 IOs, 3360785408 bytes 2022-04-10T01:35:55.456 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:9000 IOs, 3429134336 bytes 2022-04-10T01:35:55.588 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:9100 IOs, 3453095936 bytes 2022-04-10T01:35:55.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:35:55.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:35:55.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:35:55.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:35:55.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:35:55.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_44 2022-04-10T01:35:55.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image_and_enable_mirror cluster2 mirror image_44 snapshot 128M 2022-04-10T01:35:55.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:35:55.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:35:55.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:35:55.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:35:55.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_44 2022-04-10T01:35:55.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:35:55.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:35:55.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:35:55.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:35:55.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:35:55.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:35:55.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snapshot ']' 2022-04-10T01:35:55.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:35:55.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_44 128M 2022-04-10T01:35:55.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:35:55.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:35:55.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:35:55.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:35:55.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_44 2022-04-10T01:35:55.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:35:55.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:35:55.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:35:55.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:35:55.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:35:55.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:35:55.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 128M ']' 2022-04-10T01:35:55.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=128M 2022-04-10T01:35:55.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:35:55.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_44 2022-04-10T01:35:55.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' image = image ']' 2022-04-10T01:35:55.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_44 snapshot 2022-04-10T01:35:55.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:35:55.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:35:55.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_44 2022-04-10T01:35:55.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:35:55.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:35:55.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:35:55.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:35:55.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:35:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_44 snapshot 2022-04-10T01:35:56.544 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:35:56.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_44 2022-04-10T01:35:56.590 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_44': 2022-04-10T01:35:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:35:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:35:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:35:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 1403f91e30b3 2022-04-10T01:35:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.1403f91e30b3 2022-04-10T01:35:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:35:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:35:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:35:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:35:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:35:55 2022 2022-04-10T01:35:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:35:55 2022 2022-04-10T01:35:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:35:55 2022 2022-04-10T01:35:56.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:35:56.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:35:56.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 06cfc5f2-3677-4f04-83bb-d75fb95a469d 2022-04-10T01:35:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:35:56.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:35:56.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_44 2022-04-10T01:35:56.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:35:56.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:35:56.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_44 2022-04-10T01:35:56.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:35:56.600 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=7 2022-04-10T01:35:56.600 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:35:56.600 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:35:56.600 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:35:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:35:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:35:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_44 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:35:56.761 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 13138944 bytes 2022-04-10T01:35:56.917 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 58128384 bytes 2022-04-10T01:35:57.117 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 136540160 bytes 2022-04-10T01:35:57.300 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 200233984 bytes 2022-04-10T01:35:57.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 286094336 bytes 2022-04-10T01:35:57.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 349549568 bytes 2022-04-10T01:35:57.833 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 373334016 bytes 2022-04-10T01:35:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 373934080 bytes 2022-04-10T01:35:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 447930368 bytes 2022-04-10T01:35:58.335 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 521842688 bytes 2022-04-10T01:35:58.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 557811712 bytes 2022-04-10T01:35:58.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 597164032 bytes 2022-04-10T01:35:58.771 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 633476096 bytes 2022-04-10T01:35:58.999 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 708824064 bytes 2022-04-10T01:35:59.135 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 753988608 bytes 2022-04-10T01:35:59.276 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 754603008 bytes 2022-04-10T01:35:59.391 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 765655040 bytes 2022-04-10T01:35:59.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 781953024 bytes 2022-04-10T01:35:59.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 792852480 bytes 2022-04-10T01:35:59.819 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 859406336 bytes 2022-04-10T01:35:59.971 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 884045824 bytes 2022-04-10T01:36:00.175 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 971108352 bytes 2022-04-10T01:36:00.459 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 1109643264 bytes 2022-04-10T01:36:00.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 1226328064 bytes 2022-04-10T01:36:00.870 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 1265055744 bytes 2022-04-10T01:36:01.063 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1329123328 bytes 2022-04-10T01:36:01.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1382807552 bytes 2022-04-10T01:36:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1425976320 bytes 2022-04-10T01:36:01.532 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1445635072 bytes 2022-04-10T01:36:01.729 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1517083648 bytes 2022-04-10T01:36:01.867 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1541059584 bytes 2022-04-10T01:36:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1589731328 bytes 2022-04-10T01:36:02.210 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1634060288 bytes 2022-04-10T01:36:02.361 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1677100032 bytes 2022-04-10T01:36:02.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1706070016 bytes 2022-04-10T01:36:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1765393408 bytes 2022-04-10T01:36:02.892 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1832478720 bytes 2022-04-10T01:36:03.029 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1856657408 bytes 2022-04-10T01:36:03.148 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1866292224 bytes 2022-04-10T01:36:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1899219968 bytes 2022-04-10T01:36:03.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1991459840 bytes 2022-04-10T01:36:03.607 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:36:03.608 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:36:03.608 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:36:03.609 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:36:03.609 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:36:03.609 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_45 2022-04-10T01:36:03.610 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image_and_enable_mirror cluster2 mirror image_45 snapshot 128M 2022-04-10T01:36:03.610 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:36:03.610 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:36:03.610 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:36:03.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:36:03.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_45 2022-04-10T01:36:03.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:36:03.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:36:03.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:36:03.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:36:03.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:36:03.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:36:03.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snapshot ']' 2022-04-10T01:36:03.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:36:03.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_45 128M 2022-04-10T01:36:03.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:36:03.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:36:03.614 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:36:03.614 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:36:03.614 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_45 2022-04-10T01:36:03.614 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:36:03.615 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:36:03.615 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:36:03.615 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:36:03.616 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:36:03.616 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:36:03.616 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 128M ']' 2022-04-10T01:36:03.616 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=128M 2022-04-10T01:36:03.617 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:36:03.617 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_45 2022-04-10T01:36:03.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' image = image ']' 2022-04-10T01:36:03.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_45 snapshot 2022-04-10T01:36:03.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:36:03.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:36:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_45 2022-04-10T01:36:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:36:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:36:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:36:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:36:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:36:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_45 snapshot 2022-04-10T01:36:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:36:04.587 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_45 2022-04-10T01:36:04.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_45': 2022-04-10T01:36:04.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:36:04.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:36:04.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:36:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 140d6272024d 2022-04-10T01:36:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.140d6272024d 2022-04-10T01:36:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:36:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:36:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:36:04.625 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:36:04.625 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:36:03 2022 2022-04-10T01:36:04.625 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:36:03 2022 2022-04-10T01:36:04.625 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:36:03 2022 2022-04-10T01:36:04.625 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:36:04.626 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:36:04.626 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: eb158ebf-3a87-4b1d-a50e-a8e02e82428f 2022-04-10T01:36:04.626 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:36:04.628 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:36:04.628 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_45 2022-04-10T01:36:04.629 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:36:04.629 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:36:04.629 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_45 2022-04-10T01:36:04.629 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:36:04.629 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=12 2022-04-10T01:36:04.630 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:36:04.630 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:36:04.630 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:36:04.630 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:36:04.630 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:36:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_45 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:36:04.861 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 68555776 bytes 2022-04-10T01:36:04.993 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 86222848 bytes 2022-04-10T01:36:05.146 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 127337472 bytes 2022-04-10T01:36:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 179381248 bytes 2022-04-10T01:36:05.423 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 180004864 bytes 2022-04-10T01:36:05.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 207168512 bytes 2022-04-10T01:36:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 252283904 bytes 2022-04-10T01:36:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 298717184 bytes 2022-04-10T01:36:06.021 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 328922112 bytes 2022-04-10T01:36:06.175 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 364540928 bytes 2022-04-10T01:36:06.330 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 402125824 bytes 2022-04-10T01:36:06.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 425221120 bytes 2022-04-10T01:36:06.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 479184896 bytes 2022-04-10T01:36:06.856 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 553689088 bytes 2022-04-10T01:36:07.037 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 604437504 bytes 2022-04-10T01:36:07.158 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 611785728 bytes 2022-04-10T01:36:07.309 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 650981376 bytes 2022-04-10T01:36:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 718367744 bytes 2022-04-10T01:36:07.630 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 735968256 bytes 2022-04-10T01:36:07.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 796907520 bytes 2022-04-10T01:36:07.958 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 823406592 bytes 2022-04-10T01:36:08.072 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 846486528 bytes 2022-04-10T01:36:08.233 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 871813120 bytes 2022-04-10T01:36:08.395 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 903479296 bytes 2022-04-10T01:36:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 960258048 bytes 2022-04-10T01:36:08.740 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1004182528 bytes 2022-04-10T01:36:08.860 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1010604032 bytes 2022-04-10T01:36:09.073 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1089995776 bytes 2022-04-10T01:36:09.216 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1116113920 bytes 2022-04-10T01:36:09.400 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1179719680 bytes 2022-04-10T01:36:09.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1204015104 bytes 2022-04-10T01:36:09.777 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1305654272 bytes 2022-04-10T01:36:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1346713600 bytes 2022-04-10T01:36:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1347318784 bytes 2022-04-10T01:36:10.227 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1403881472 bytes 2022-04-10T01:36:10.274 INFO:tasks.ceph.cluster2.osd.2.smithi043.stderr:2022-04-10T01:36:10.271+0000 7f8be0453700 -1 reset not still connected to 0x55f2a4637ee0 2022-04-10T01:36:10.275 INFO:tasks.ceph.cluster2.osd.2.smithi043.stderr:2022-04-10T01:36:10.271+0000 7f8be0453700 -1 reset not still connected to 0x55f2b9563040 2022-04-10T01:36:10.275 INFO:tasks.ceph.cluster2.osd.2.smithi043.stderr:2022-04-10T01:36:10.271+0000 7f8be0453700 -1 reset not still connected to 0x55f2bab09d40 2022-04-10T01:36:10.275 INFO:tasks.ceph.cluster2.osd.2.smithi043.stderr:2022-04-10T01:36:10.271+0000 7f8be0453700 -1 reset not still connected to 0x55f2bbb6e750 2022-04-10T01:36:10.473 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1461126144 bytes 2022-04-10T01:36:10.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1489636352 bytes 2022-04-10T01:36:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1563156480 bytes 2022-04-10T01:36:11.102 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1648313344 bytes 2022-04-10T01:36:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1648930816 bytes 2022-04-10T01:36:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1688693760 bytes 2022-04-10T01:36:11.541 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 1734758400 bytes 2022-04-10T01:36:11.750 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 1807175680 bytes 2022-04-10T01:36:11.867 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 1818244096 bytes 2022-04-10T01:36:12.017 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 1847746560 bytes 2022-04-10T01:36:12.152 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 1864438784 bytes 2022-04-10T01:36:12.275 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 1873510400 bytes 2022-04-10T01:36:12.442 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 1923932160 bytes 2022-04-10T01:36:12.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 1966163968 bytes 2022-04-10T01:36:12.751 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 1993472000 bytes 2022-04-10T01:36:12.876 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 2010631168 bytes 2022-04-10T01:36:13.063 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 2075160576 bytes 2022-04-10T01:36:13.223 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 2107419648 bytes 2022-04-10T01:36:13.359 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5400 IOs, 2140573696 bytes 2022-04-10T01:36:13.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5500 IOs, 2198079488 bytes 2022-04-10T01:36:13.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5600 IOs, 2212149248 bytes 2022-04-10T01:36:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5700 IOs, 2288105472 bytes 2022-04-10T01:36:14.002 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5800 IOs, 2294305792 bytes 2022-04-10T01:36:14.275 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5900 IOs, 2413937664 bytes 2022-04-10T01:36:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6000 IOs, 2463166464 bytes 2022-04-10T01:36:14.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6100 IOs, 2542476288 bytes 2022-04-10T01:36:14.759 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6200 IOs, 2547773440 bytes 2022-04-10T01:36:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6300 IOs, 2592185344 bytes 2022-04-10T01:36:15.051 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6400 IOs, 2616508416 bytes 2022-04-10T01:36:15.226 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6500 IOs, 2672881664 bytes 2022-04-10T01:36:15.395 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6600 IOs, 2718194688 bytes 2022-04-10T01:36:15.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6700 IOs, 2774448128 bytes 2022-04-10T01:36:15.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6800 IOs, 2777947136 bytes 2022-04-10T01:36:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6900 IOs, 2798807040 bytes 2022-04-10T01:36:15.946 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7000 IOs, 2815590400 bytes 2022-04-10T01:36:16.064 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7100 IOs, 2829496320 bytes 2022-04-10T01:36:16.217 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7200 IOs, 2860522496 bytes 2022-04-10T01:36:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7300 IOs, 2949194752 bytes 2022-04-10T01:36:16.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7400 IOs, 2988929024 bytes 2022-04-10T01:36:16.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:36:16.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:36:16.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:36:16.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:36:16.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:36:16.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_46 2022-04-10T01:36:16.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image_and_enable_mirror cluster2 mirror image_46 snapshot 128M 2022-04-10T01:36:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:36:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:36:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:36:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:36:16.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_46 2022-04-10T01:36:16.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:36:16.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:36:16.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:36:16.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:36:16.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:36:16.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:36:16.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snapshot ']' 2022-04-10T01:36:16.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:36:16.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_46 128M 2022-04-10T01:36:16.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:36:16.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:36:16.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:36:16.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:36:16.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_46 2022-04-10T01:36:16.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:36:16.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:36:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:36:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:36:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:36:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:36:16.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 128M ']' 2022-04-10T01:36:16.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=128M 2022-04-10T01:36:16.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:36:16.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_46 2022-04-10T01:36:16.889 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' image = image ']' 2022-04-10T01:36:16.890 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_46 snapshot 2022-04-10T01:36:16.890 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:36:16.890 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:36:16.890 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_46 2022-04-10T01:36:16.890 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:36:16.891 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:36:16.891 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:36:16.891 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:36:16.891 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:36:16.892 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_46 snapshot 2022-04-10T01:36:17.631 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:36:17.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_46 2022-04-10T01:36:17.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_46': 2022-04-10T01:36:17.677 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:36:17.677 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:36:17.677 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:36:17.677 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 14166145b22e 2022-04-10T01:36:17.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.14166145b22e 2022-04-10T01:36:17.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:36:17.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:36:17.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:36:17.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:36:17.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:36:16 2022 2022-04-10T01:36:17.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:36:16 2022 2022-04-10T01:36:17.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:36:16 2022 2022-04-10T01:36:17.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:36:17.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:36:17.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: c83a7972-17a4-4b3c-b33c-85a8de6039e5 2022-04-10T01:36:17.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:36:17.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:36:17.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_46 2022-04-10T01:36:17.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:36:17.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:36:17.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_46 2022-04-10T01:36:17.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:36:17.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=14 2022-04-10T01:36:17.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:36:17.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:36:17.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:36:17.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:36:17.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:36:17.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_46 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:36:17.884 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 46826496 bytes 2022-04-10T01:36:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 75741184 bytes 2022-04-10T01:36:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 80789504 bytes 2022-04-10T01:36:18.289 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 133109760 bytes 2022-04-10T01:36:18.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 201577472 bytes 2022-04-10T01:36:18.601 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 218113024 bytes 2022-04-10T01:36:18.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 300815360 bytes 2022-04-10T01:36:18.919 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 324354048 bytes 2022-04-10T01:36:19.168 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 426365952 bytes 2022-04-10T01:36:19.286 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 426978304 bytes 2022-04-10T01:36:19.453 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 476917760 bytes 2022-04-10T01:36:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 543130624 bytes 2022-04-10T01:36:19.819 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 599794688 bytes 2022-04-10T01:36:19.996 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 658321408 bytes 2022-04-10T01:36:20.152 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 699505664 bytes 2022-04-10T01:36:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 786041856 bytes 2022-04-10T01:36:20.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 799704064 bytes 2022-04-10T01:36:20.658 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 868076544 bytes 2022-04-10T01:36:20.783 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 885491712 bytes 2022-04-10T01:36:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 896022528 bytes 2022-04-10T01:36:21.066 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 950094848 bytes 2022-04-10T01:36:21.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 966426624 bytes 2022-04-10T01:36:21.318 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 1006546944 bytes 2022-04-10T01:36:21.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 1027422208 bytes 2022-04-10T01:36:21.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 1052384256 bytes 2022-04-10T01:36:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1078780928 bytes 2022-04-10T01:36:21.921 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1124373504 bytes 2022-04-10T01:36:22.139 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1227272192 bytes 2022-04-10T01:36:22.293 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1262902272 bytes 2022-04-10T01:36:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1317669888 bytes 2022-04-10T01:36:22.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1337711616 bytes 2022-04-10T01:36:22.770 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1392468992 bytes 2022-04-10T01:36:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1438537728 bytes 2022-04-10T01:36:23.089 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1480350720 bytes 2022-04-10T01:36:23.283 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1553832960 bytes 2022-04-10T01:36:23.457 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1619286016 bytes 2022-04-10T01:36:23.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1619905536 bytes 2022-04-10T01:36:23.772 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1694528512 bytes 2022-04-10T01:36:23.910 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1727736832 bytes 2022-04-10T01:36:24.021 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1733581824 bytes 2022-04-10T01:36:24.260 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1835901952 bytes 2022-04-10T01:36:24.408 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 1868181504 bytes 2022-04-10T01:36:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 1903410176 bytes 2022-04-10T01:36:24.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 1929984000 bytes 2022-04-10T01:36:24.855 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 1973022720 bytes 2022-04-10T01:36:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 2032309248 bytes 2022-04-10T01:36:25.159 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 2060779520 bytes 2022-04-10T01:36:25.310 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 2092968960 bytes 2022-04-10T01:36:25.427 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 2111966208 bytes 2022-04-10T01:36:25.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 2164305920 bytes 2022-04-10T01:36:25.762 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 2206013440 bytes 2022-04-10T01:36:25.894 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 2223733760 bytes 2022-04-10T01:36:26.093 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 2301293568 bytes 2022-04-10T01:36:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5400 IOs, 2329531392 bytes 2022-04-10T01:36:26.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5500 IOs, 2361511936 bytes 2022-04-10T01:36:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5600 IOs, 2405750784 bytes 2022-04-10T01:36:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5700 IOs, 2429298688 bytes 2022-04-10T01:36:26.884 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5800 IOs, 2525043712 bytes 2022-04-10T01:36:27.078 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5900 IOs, 2588547072 bytes 2022-04-10T01:36:27.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6000 IOs, 2638525440 bytes 2022-04-10T01:36:27.395 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6100 IOs, 2677189632 bytes 2022-04-10T01:36:27.566 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6200 IOs, 2745218048 bytes 2022-04-10T01:36:27.731 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6300 IOs, 2784931840 bytes 2022-04-10T01:36:27.862 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6400 IOs, 2804032512 bytes 2022-04-10T01:36:28.013 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6500 IOs, 2845234176 bytes 2022-04-10T01:36:28.129 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6600 IOs, 2851475456 bytes 2022-04-10T01:36:28.297 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6700 IOs, 2904403968 bytes 2022-04-10T01:36:28.416 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6800 IOs, 2917152768 bytes 2022-04-10T01:36:28.606 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6900 IOs, 2980637696 bytes 2022-04-10T01:36:28.776 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7000 IOs, 3027848192 bytes 2022-04-10T01:36:28.925 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7100 IOs, 3061523456 bytes 2022-04-10T01:36:29.068 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7200 IOs, 3096879104 bytes 2022-04-10T01:36:29.151 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8371498816756114) > weight (0.25) 2022-04-10T01:36:29.151 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.31061144492021264) > weight (0.25) 2022-04-10T01:36:29.152 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2022-04-10T01:36:29.152 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2022-04-10T01:36:29.153 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2022-04-10T01:36:29.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi043.stderr:2022-04-10T01:36:29.151+0000 7f247d184700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 51096) UID: 1000 2022-04-10T01:36:29.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7300 IOs, 3147992064 bytes 2022-04-10T01:36:29.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi043.stderr:daemon-helper: command failed with exit status 1 2022-04-10T01:36:29.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7400 IOs, 3150313472 bytes 2022-04-10T01:36:29.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7500 IOs, 3169157120 bytes 2022-04-10T01:36:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7600 IOs, 3227155456 bytes 2022-04-10T01:36:29.853 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7700 IOs, 3270875136 bytes 2022-04-10T01:36:29.959 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7800 IOs, 3282262016 bytes 2022-04-10T01:36:30.154 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2022-04-10T01:36:30.155 INFO:teuthology.orchestra.run:waiting for 600 2022-04-10T01:36:30.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-04-10T01:36:30.156 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2022-04-10T01:36:30.156 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2022-04-10T01:36:30.157 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2022-04-10T01:36:30.159 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2022-04-10T01:36:30.160 INFO:tasks.rbd_mirror_thrash:waiting for 104 secs before thrashing 2022-04-10T01:36:30.160 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7900 IOs, 3347392512 bytes 2022-04-10T01:36:30.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8000 IOs, 3382219776 bytes 2022-04-10T01:36:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8100 IOs, 3470302208 bytes 2022-04-10T01:36:30.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8200 IOs, 3486750720 bytes 2022-04-10T01:36:30.849 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8300 IOs, 3502299136 bytes 2022-04-10T01:36:31.047 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8400 IOs, 3577977856 bytes 2022-04-10T01:36:31.196 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8500 IOs, 3608733696 bytes 2022-04-10T01:36:31.390 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8600 IOs, 3686130688 bytes 2022-04-10T01:36:31.592 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8700 IOs, 3755915264 bytes 2022-04-10T01:36:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:36:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:36:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:36:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:36:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:36:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_47 2022-04-10T01:36:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image_and_enable_mirror cluster2 mirror image_47 snapshot 128M 2022-04-10T01:36:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:36:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:36:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:36:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:36:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_47 2022-04-10T01:36:31.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:36:31.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:36:31.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:36:31.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:36:31.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:36:31.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:36:31.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snapshot ']' 2022-04-10T01:36:31.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:36:31.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_47 128M 2022-04-10T01:36:31.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:36:31.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:36:31.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:36:31.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:36:31.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_47 2022-04-10T01:36:31.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:36:31.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:36:31.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:36:31.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:36:31.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:36:31.698 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:36:31.698 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 128M ']' 2022-04-10T01:36:31.698 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=128M 2022-04-10T01:36:31.698 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:36:31.699 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_47 2022-04-10T01:36:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' image = image ']' 2022-04-10T01:36:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_47 snapshot 2022-04-10T01:36:31.732 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:36:31.732 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:36:31.732 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_47 2022-04-10T01:36:31.732 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:36:31.733 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:36:31.733 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:36:31.733 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:36:31.734 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:36:31.734 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_47 snapshot 2022-04-10T01:36:32.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:36:32.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_47 2022-04-10T01:36:32.713 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_47': 2022-04-10T01:36:32.713 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:36:32.713 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:36:32.714 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:36:32.714 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 1429801ece2e 2022-04-10T01:36:32.714 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.1429801ece2e 2022-04-10T01:36:32.714 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:36:32.715 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:36:32.715 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:36:32.715 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:36:32.715 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:36:31 2022 2022-04-10T01:36:32.716 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:36:31 2022 2022-04-10T01:36:32.716 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:36:31 2022 2022-04-10T01:36:32.716 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:36:32.716 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:36:32.717 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: a7fe8d55-51fd-4578-8dbb-89aefafe026d 2022-04-10T01:36:32.717 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:36:32.719 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:36:32.719 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_47 2022-04-10T01:36:32.719 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:36:32.719 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:36:32.720 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_47 2022-04-10T01:36:32.720 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:36:32.720 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=12 2022-04-10T01:36:32.720 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:36:32.720 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:36:32.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:36:32.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:36:32.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:36:32.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_47 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:36:32.927 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 59987968 bytes 2022-04-10T01:36:33.097 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 110134272 bytes 2022-04-10T01:36:33.264 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 164728832 bytes 2022-04-10T01:36:33.394 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 182830080 bytes 2022-04-10T01:36:33.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 220425216 bytes 2022-04-10T01:36:33.681 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 249771008 bytes 2022-04-10T01:36:33.812 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 282461184 bytes 2022-04-10T01:36:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 301152256 bytes 2022-04-10T01:36:34.094 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 333002752 bytes 2022-04-10T01:36:34.233 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 357517312 bytes 2022-04-10T01:36:34.438 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 435081216 bytes 2022-04-10T01:36:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 466361344 bytes 2022-04-10T01:36:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 477721600 bytes 2022-04-10T01:36:34.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 485236736 bytes 2022-04-10T01:36:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 535023616 bytes 2022-04-10T01:36:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 582172672 bytes 2022-04-10T01:36:35.276 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 600744960 bytes 2022-04-10T01:36:35.426 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 633006080 bytes 2022-04-10T01:36:35.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 724444160 bytes 2022-04-10T01:36:35.785 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 760845312 bytes 2022-04-10T01:36:35.942 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 800952320 bytes 2022-04-10T01:36:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 821921792 bytes 2022-04-10T01:36:36.218 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 860610560 bytes 2022-04-10T01:36:36.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 881780736 bytes 2022-04-10T01:36:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 916558848 bytes 2022-04-10T01:36:36.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 972898304 bytes 2022-04-10T01:36:36.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1007636480 bytes 2022-04-10T01:36:36.999 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1069221888 bytes 2022-04-10T01:36:37.217 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1165661184 bytes 2022-04-10T01:36:37.352 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1208673280 bytes 2022-04-10T01:36:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1234678784 bytes 2022-04-10T01:36:37.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1260269568 bytes 2022-04-10T01:36:37.770 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1266613248 bytes 2022-04-10T01:36:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1288830976 bytes 2022-04-10T01:36:38.032 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1327292416 bytes 2022-04-10T01:36:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1402463232 bytes 2022-04-10T01:36:38.385 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1423828992 bytes 2022-04-10T01:36:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1439427584 bytes 2022-04-10T01:36:38.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1468786688 bytes 2022-04-10T01:36:38.775 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1482032128 bytes 2022-04-10T01:36:38.950 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1526683648 bytes 2022-04-10T01:36:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 1543877632 bytes 2022-04-10T01:36:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 1554289664 bytes 2022-04-10T01:36:39.310 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 1569757184 bytes 2022-04-10T01:36:39.465 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 1625820160 bytes 2022-04-10T01:36:39.589 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 1636195328 bytes 2022-04-10T01:36:39.728 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 1661420544 bytes 2022-04-10T01:36:39.856 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 1681424384 bytes 2022-04-10T01:36:40.028 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 1706641408 bytes 2022-04-10T01:36:40.285 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 1823940608 bytes 2022-04-10T01:36:40.401 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 1827224576 bytes 2022-04-10T01:36:40.625 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 1874041856 bytes 2022-04-10T01:36:40.858 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 1973080064 bytes 2022-04-10T01:36:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5400 IOs, 2021252096 bytes 2022-04-10T01:36:41.143 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5500 IOs, 2028576768 bytes 2022-04-10T01:36:41.677 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5600 IOs, 2080926720 bytes 2022-04-10T01:36:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5700 IOs, 2084103168 bytes 2022-04-10T01:36:41.982 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5800 IOs, 2151675904 bytes 2022-04-10T01:36:42.121 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5900 IOs, 2180734976 bytes 2022-04-10T01:36:42.269 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6000 IOs, 2208128000 bytes 2022-04-10T01:36:42.426 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6100 IOs, 2246276096 bytes 2022-04-10T01:36:42.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6200 IOs, 2246856704 bytes 2022-04-10T01:36:42.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6300 IOs, 2272235520 bytes 2022-04-10T01:36:42.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6400 IOs, 2283765760 bytes 2022-04-10T01:36:42.967 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6500 IOs, 2326280192 bytes 2022-04-10T01:36:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6600 IOs, 2391712768 bytes 2022-04-10T01:36:43.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6700 IOs, 2478683136 bytes 2022-04-10T01:36:43.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6800 IOs, 2483345408 bytes 2022-04-10T01:36:43.870 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6900 IOs, 2541698048 bytes 2022-04-10T01:36:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7000 IOs, 2561000448 bytes 2022-04-10T01:36:44.151 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7100 IOs, 2599000064 bytes 2022-04-10T01:36:44.306 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7200 IOs, 2629352448 bytes 2022-04-10T01:36:44.447 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7300 IOs, 2662730752 bytes 2022-04-10T01:36:44.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7400 IOs, 2750061568 bytes 2022-04-10T01:36:44.725 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:36:44.725 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:36:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:36:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:36:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:36:44.726 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_48 2022-04-10T01:36:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image_and_enable_mirror cluster2 mirror image_48 snapshot 128M 2022-04-10T01:36:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:36:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:36:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:36:44.728 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:36:44.728 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_48 2022-04-10T01:36:44.728 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:36:44.728 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:36:44.728 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:36:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:36:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:36:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:36:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snapshot ']' 2022-04-10T01:36:44.730 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:36:44.730 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_48 128M 2022-04-10T01:36:44.730 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:36:44.730 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:36:44.731 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:36:44.731 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:36:44.731 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_48 2022-04-10T01:36:44.731 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:36:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:36:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:36:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:36:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:36:44.733 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:36:44.733 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 128M ']' 2022-04-10T01:36:44.733 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=128M 2022-04-10T01:36:44.733 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:36:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_48 2022-04-10T01:36:44.769 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' image = image ']' 2022-04-10T01:36:44.769 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_48 snapshot 2022-04-10T01:36:44.770 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:36:44.770 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:36:44.770 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_48 2022-04-10T01:36:44.770 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:36:44.771 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:36:44.771 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:36:44.771 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:36:44.771 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:36:44.772 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_48 snapshot 2022-04-10T01:36:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:36:45.717 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_48 2022-04-10T01:36:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_48': 2022-04-10T01:36:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:36:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:36:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:36:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 14315bf2730a 2022-04-10T01:36:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.14315bf2730a 2022-04-10T01:36:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:36:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:36:45.755 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:36:45.755 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:36:45.755 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:36:44 2022 2022-04-10T01:36:45.755 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:36:44 2022 2022-04-10T01:36:45.756 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:36:44 2022 2022-04-10T01:36:45.756 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:36:45.756 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:36:45.756 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: dc4e02d6-0aa7-4947-95ef-025b13fdc64d 2022-04-10T01:36:45.756 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:36:45.758 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:36:45.759 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_48 2022-04-10T01:36:45.759 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:36:45.759 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:36:45.760 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_48 2022-04-10T01:36:45.760 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:36:45.760 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=10 2022-04-10T01:36:45.760 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:36:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:36:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:36:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:36:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:36:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_48 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:36:45.968 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 47053824 bytes 2022-04-10T01:36:46.122 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 88376320 bytes 2022-04-10T01:36:46.289 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 137226240 bytes 2022-04-10T01:36:46.455 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 189620224 bytes 2022-04-10T01:36:46.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 205939712 bytes 2022-04-10T01:36:46.738 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 240518144 bytes 2022-04-10T01:36:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 278104064 bytes 2022-04-10T01:36:47.003 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 290707456 bytes 2022-04-10T01:36:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 328169472 bytes 2022-04-10T01:36:47.365 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 413960192 bytes 2022-04-10T01:36:47.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 457720832 bytes 2022-04-10T01:36:47.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 487448576 bytes 2022-04-10T01:36:47.849 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 540710912 bytes 2022-04-10T01:36:47.997 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 581198848 bytes 2022-04-10T01:36:48.119 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 595126272 bytes 2022-04-10T01:36:48.307 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 665763840 bytes 2022-04-10T01:36:48.450 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 706282496 bytes 2022-04-10T01:36:48.627 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 757147648 bytes 2022-04-10T01:36:48.747 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 761533440 bytes 2022-04-10T01:36:48.926 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 821000192 bytes 2022-04-10T01:36:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 846090240 bytes 2022-04-10T01:36:49.181 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 874779648 bytes 2022-04-10T01:36:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 898257920 bytes 2022-04-10T01:36:49.528 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 961492992 bytes 2022-04-10T01:36:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 1004113920 bytes 2022-04-10T01:36:49.868 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1064892416 bytes 2022-04-10T01:36:49.993 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1078287360 bytes 2022-04-10T01:36:50.166 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1127653376 bytes 2022-04-10T01:36:50.332 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1176778752 bytes 2022-04-10T01:36:50.491 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1229180928 bytes 2022-04-10T01:36:50.632 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1251815424 bytes 2022-04-10T01:36:50.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1296038912 bytes 2022-04-10T01:36:50.919 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1303326720 bytes 2022-04-10T01:36:51.021 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1303958528 bytes 2022-04-10T01:36:51.209 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1368979456 bytes 2022-04-10T01:36:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1388691456 bytes 2022-04-10T01:36:51.469 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1404964864 bytes 2022-04-10T01:36:51.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1479070720 bytes 2022-04-10T01:36:51.796 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1496549376 bytes 2022-04-10T01:36:51.907 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1497935872 bytes 2022-04-10T01:36:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1556894720 bytes 2022-04-10T01:36:52.276 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 1620953088 bytes 2022-04-10T01:36:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 1692596224 bytes 2022-04-10T01:36:52.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 1780031488 bytes 2022-04-10T01:36:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 1799812096 bytes 2022-04-10T01:36:52.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 1802874880 bytes 2022-04-10T01:36:53.107 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 1860269056 bytes 2022-04-10T01:36:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 1912716288 bytes 2022-04-10T01:36:53.435 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 1944500224 bytes 2022-04-10T01:36:53.583 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 1980190720 bytes 2022-04-10T01:36:53.732 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 2014446592 bytes 2022-04-10T01:36:53.906 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 2071605248 bytes 2022-04-10T01:36:54.027 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 2081381376 bytes 2022-04-10T01:36:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5400 IOs, 2235918336 bytes 2022-04-10T01:36:54.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5500 IOs, 2304944128 bytes 2022-04-10T01:36:54.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5600 IOs, 2305568768 bytes 2022-04-10T01:36:54.761 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5700 IOs, 2340419584 bytes 2022-04-10T01:36:54.988 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5800 IOs, 2437700608 bytes 2022-04-10T01:36:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5900 IOs, 2522571776 bytes 2022-04-10T01:36:55.297 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6000 IOs, 2525430784 bytes 2022-04-10T01:36:55.453 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6100 IOs, 2572318720 bytes 2022-04-10T01:36:55.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6200 IOs, 2650285056 bytes 2022-04-10T01:36:55.765 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:36:55.765 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:36:55.765 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:36:55.766 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:36:55.766 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:36:55.766 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_49 2022-04-10T01:36:55.766 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image_and_enable_mirror cluster2 mirror image_49 snapshot 128M 2022-04-10T01:36:55.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:36:55.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:36:55.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:36:55.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:36:55.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_49 2022-04-10T01:36:55.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:36:55.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:36:55.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:36:55.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:36:55.769 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:36:55.769 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:36:55.769 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snapshot ']' 2022-04-10T01:36:55.769 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:36:55.770 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_49 128M 2022-04-10T01:36:55.770 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:36:55.770 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:36:55.770 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:36:55.771 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:36:55.771 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_49 2022-04-10T01:36:55.771 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:36:55.771 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:36:55.772 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:36:55.772 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:36:55.772 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:36:55.772 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:36:55.772 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 128M ']' 2022-04-10T01:36:55.773 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=128M 2022-04-10T01:36:55.773 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:36:55.773 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_49 2022-04-10T01:36:55.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' image = image ']' 2022-04-10T01:36:55.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_49 snapshot 2022-04-10T01:36:55.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:36:55.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:36:55.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_49 2022-04-10T01:36:55.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:36:55.816 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:36:55.816 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:36:55.816 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:36:55.816 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:36:55.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_49 snapshot 2022-04-10T01:36:55.843 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2022-04-10T01:36:55.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2022-04-10T01:36:55.844 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2022-04-10T01:36:55.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi043.stderr:2022-04-10T01:36:55.843+0000 7ff5bf4b7700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 52517) UID: 1000 2022-04-10T01:36:56.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi043.stderr:daemon-helper: command failed with exit status 1 2022-04-10T01:36:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:36:56.754 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_49 2022-04-10T01:36:56.789 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_49': 2022-04-10T01:36:56.789 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:36:56.790 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:36:56.790 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:36:56.790 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 143ba2ad202d 2022-04-10T01:36:56.790 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.143ba2ad202d 2022-04-10T01:36:56.791 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:36:56.791 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:36:56.791 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:36:56.791 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:36:56.791 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:36:55 2022 2022-04-10T01:36:56.792 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:36:55 2022 2022-04-10T01:36:56.792 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:36:55 2022 2022-04-10T01:36:56.792 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:36:56.792 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:36:56.793 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: dadd4ce2-bc41-4878-8215-b7901a7d9b7c 2022-04-10T01:36:56.793 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:36:56.794 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:36:56.795 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_49 2022-04-10T01:36:56.795 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:36:56.795 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:36:56.796 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_49 2022-04-10T01:36:56.796 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:36:56.796 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=13 2022-04-10T01:36:56.796 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:36:56.797 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:36:56.797 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:36:56.797 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:36:56.797 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:36:56.798 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_49 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:36:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 92235776 bytes 2022-04-10T01:36:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 120402944 bytes 2022-04-10T01:36:57.327 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 150261760 bytes 2022-04-10T01:36:57.464 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 177801216 bytes 2022-04-10T01:36:57.606 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 208312320 bytes 2022-04-10T01:36:57.811 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 287889408 bytes 2022-04-10T01:36:57.947 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 309680128 bytes 2022-04-10T01:36:58.166 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 403772416 bytes 2022-04-10T01:36:58.288 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 412717056 bytes 2022-04-10T01:36:58.413 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 429650944 bytes 2022-04-10T01:36:58.542 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 452131840 bytes 2022-04-10T01:36:58.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 467265536 bytes 2022-04-10T01:36:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 532401152 bytes 2022-04-10T01:36:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 558067712 bytes 2022-04-10T01:36:59.128 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 593287168 bytes 2022-04-10T01:36:59.293 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 627714048 bytes 2022-04-10T01:36:59.465 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 681325568 bytes 2022-04-10T01:36:59.647 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 743162880 bytes 2022-04-10T01:36:59.754 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 752720896 bytes 2022-04-10T01:36:59.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 810692608 bytes 2022-04-10T01:37:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 861600768 bytes 2022-04-10T01:37:00.297 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 937868288 bytes 2022-04-10T01:37:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 984349696 bytes 2022-04-10T01:37:00.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 1065325568 bytes 2022-04-10T01:37:00.844 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 1118658560 bytes 2022-04-10T01:37:01.067 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1222775808 bytes 2022-04-10T01:37:01.231 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1267396608 bytes 2022-04-10T01:37:01.398 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1314108416 bytes 2022-04-10T01:37:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1351626752 bytes 2022-04-10T01:37:01.702 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1395458048 bytes 2022-04-10T01:37:01.828 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1408751616 bytes 2022-04-10T01:37:01.845 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2022-04-10T01:37:01.845 INFO:teuthology.orchestra.run:waiting for 600 2022-04-10T01:37:01.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-04-10T01:37:01.846 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2022-04-10T01:37:01.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2022-04-10T01:37:01.847 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2022-04-10T01:37:01.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2022-04-10T01:37:01.850 INFO:tasks.rbd_mirror_thrash:waiting for 87 secs before thrashing 2022-04-10T01:37:01.952 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1415410688 bytes 2022-04-10T01:37:02.115 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1448625152 bytes 2022-04-10T01:37:02.398 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1587781632 bytes 2022-04-10T01:37:02.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1661925376 bytes 2022-04-10T01:37:02.763 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1712094208 bytes 2022-04-10T01:37:02.937 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1762320384 bytes 2022-04-10T01:37:03.074 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1789941760 bytes 2022-04-10T01:37:03.249 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1849883648 bytes 2022-04-10T01:37:03.455 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1918795776 bytes 2022-04-10T01:37:03.609 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1961183232 bytes 2022-04-10T01:37:03.758 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 2001492992 bytes 2022-04-10T01:37:03.884 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 2025027584 bytes 2022-04-10T01:37:04.015 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 2037811200 bytes 2022-04-10T01:37:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 2064496640 bytes 2022-04-10T01:37:04.324 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 2116878336 bytes 2022-04-10T01:37:04.492 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 2178646016 bytes 2022-04-10T01:37:04.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 2179256320 bytes 2022-04-10T01:37:04.759 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 2212652032 bytes 2022-04-10T01:37:04.876 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 2227816448 bytes 2022-04-10T01:37:05.057 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 2284514304 bytes 2022-04-10T01:37:05.159 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 2285130752 bytes 2022-04-10T01:37:05.319 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 2331522048 bytes 2022-04-10T01:37:05.443 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5400 IOs, 2350792704 bytes 2022-04-10T01:37:05.621 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5500 IOs, 2403820544 bytes 2022-04-10T01:37:05.810 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5600 IOs, 2471621632 bytes 2022-04-10T01:37:05.954 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5700 IOs, 2502633472 bytes 2022-04-10T01:37:06.155 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5800 IOs, 2585401344 bytes 2022-04-10T01:37:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5900 IOs, 2607927296 bytes 2022-04-10T01:37:06.460 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6000 IOs, 2656676864 bytes 2022-04-10T01:37:06.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6100 IOs, 2665009152 bytes 2022-04-10T01:37:06.702 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6200 IOs, 2680999936 bytes 2022-04-10T01:37:06.863 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6300 IOs, 2728628224 bytes 2022-04-10T01:37:07.024 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6400 IOs, 2777011200 bytes 2022-04-10T01:37:07.161 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6500 IOs, 2801336320 bytes 2022-04-10T01:37:07.296 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6600 IOs, 2843765760 bytes 2022-04-10T01:37:07.467 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6700 IOs, 2868195328 bytes 2022-04-10T01:37:07.602 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6800 IOs, 2891886592 bytes 2022-04-10T01:37:07.740 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6900 IOs, 2935798784 bytes 2022-04-10T01:37:07.918 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7000 IOs, 2975056896 bytes 2022-04-10T01:37:08.065 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7100 IOs, 3010578432 bytes 2022-04-10T01:37:08.270 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7200 IOs, 3085627392 bytes 2022-04-10T01:37:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7300 IOs, 3133469696 bytes 2022-04-10T01:37:08.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7400 IOs, 3163141120 bytes 2022-04-10T01:37:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7500 IOs, 3188627456 bytes 2022-04-10T01:37:08.886 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7600 IOs, 3239866368 bytes 2022-04-10T01:37:09.055 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7700 IOs, 3286239232 bytes 2022-04-10T01:37:09.216 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7800 IOs, 3344635904 bytes 2022-04-10T01:37:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7900 IOs, 3447238656 bytes 2022-04-10T01:37:09.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8000 IOs, 3485307904 bytes 2022-04-10T01:37:09.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:37:09.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:37:09.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:37:09.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:37:09.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:37:09.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_50 2022-04-10T01:37:09.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image_and_enable_mirror cluster2 mirror image_50 snapshot 128M 2022-04-10T01:37:09.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:37:09.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:37:09.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:37:09.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:37:09.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_50 2022-04-10T01:37:09.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:37:09.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:37:09.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:37:09.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:37:09.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:37:09.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:37:09.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n snapshot ']' 2022-04-10T01:37:09.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:37:09.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_50 128M 2022-04-10T01:37:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:37:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:37:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:37:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:37:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_50 2022-04-10T01:37:09.809 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:37:09.809 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:37:09.809 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:37:09.809 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:37:09.810 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:37:09.810 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:37:09.810 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 128M ']' 2022-04-10T01:37:09.810 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=128M 2022-04-10T01:37:09.811 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:37:09.811 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_50 2022-04-10T01:37:09.845 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' image = image ']' 2022-04-10T01:37:09.846 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_50 snapshot 2022-04-10T01:37:09.846 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:37:09.846 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:37:09.846 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_50 2022-04-10T01:37:09.847 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:37:09.847 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:37:09.847 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:37:09.847 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:37:09.847 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:37:09.848 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_50 snapshot 2022-04-10T01:37:10.810 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:37:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_50 2022-04-10T01:37:10.856 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_50': 2022-04-10T01:37:10.856 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:37:10.856 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:37:10.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:37:10.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 144af618ee2e 2022-04-10T01:37:10.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.144af618ee2e 2022-04-10T01:37:10.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:37:10.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:37:10.858 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:37:10.858 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:37:10.858 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:37:09 2022 2022-04-10T01:37:10.858 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:37:09 2022 2022-04-10T01:37:10.859 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:37:09 2022 2022-04-10T01:37:10.859 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:37:10.859 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:37:10.859 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: a17938e9-2a1e-432d-9eca-f1aacf780959 2022-04-10T01:37:10.860 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:37:10.861 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n '' ']' 2022-04-10T01:37:10.862 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_50 2022-04-10T01:37:10.862 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:37:10.862 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:37:10.862 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_50 2022-04-10T01:37:10.863 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:37:10.863 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=5 2022-04-10T01:37:10.863 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:37:10.863 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:37:10.864 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:37:10.864 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:37:10.864 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:37:10.864 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_50 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:37:11.103 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 60758016 bytes 2022-04-10T01:37:11.702 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 142375936 bytes 2022-04-10T01:37:11.858 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 196159488 bytes 2022-04-10T01:37:11.990 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 212196352 bytes 2022-04-10T01:37:12.180 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 291435520 bytes 2022-04-10T01:37:12.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 355662848 bytes 2022-04-10T01:37:12.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 445470720 bytes 2022-04-10T01:37:12.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 522908672 bytes 2022-04-10T01:37:12.945 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 577782784 bytes 2022-04-10T01:37:13.131 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 634015744 bytes 2022-04-10T01:37:13.248 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 646279168 bytes 2022-04-10T01:37:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 699324416 bytes 2022-04-10T01:37:13.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 731098112 bytes 2022-04-10T01:37:13.707 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 756436992 bytes 2022-04-10T01:37:13.867 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 805066752 bytes 2022-04-10T01:37:14.004 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 831608832 bytes 2022-04-10T01:37:14.125 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 851957760 bytes 2022-04-10T01:37:14.362 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 942629888 bytes 2022-04-10T01:37:14.562 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 1018146816 bytes 2022-04-10T01:37:14.758 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 1095326720 bytes 2022-04-10T01:37:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 1112071168 bytes 2022-04-10T01:37:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 1115192320 bytes 2022-04-10T01:37:15.174 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 1166051328 bytes 2022-04-10T01:37:15.284 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 1183298560 bytes 2022-04-10T01:37:15.429 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 1197705216 bytes 2022-04-10T01:37:15.609 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1258309632 bytes 2022-04-10T01:37:15.741 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1284518912 bytes 2022-04-10T01:37:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:37:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:37:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:37:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:37:15.870 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_pool_images cluster2 mirror 50 2022-04-10T01:37:15.870 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:37:15.870 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:37:15.870 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_count=50 2022-04-10T01:37:15.871 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:37:15.871 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local count 2022-04-10T01:37:15.871 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local last_count=0 2022-04-10T01:37:15.871 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:37:15.872 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ seq 1 40 2022-04-10T01:37:15.873 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in `seq 1 40` 2022-04-10T01:37:15.873 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 1 -ne 1 2022-04-10T01:37:15.873 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2022-04-10T01:37:15.873 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep 'images: ' 2022-04-10T01:37:15.874 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cut '-d ' -f 2 2022-04-10T01:37:15.903 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: failed to query services: (13) Permission denied 2022-04-10T01:37:15.907 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ count=50 2022-04-10T01:37:15.907 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 50 = 50 2022-04-10T01:37:15.907 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:37:15.907 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_pool_healthy cluster2 mirror 2022-04-10T01:37:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:37:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:37:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:37:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state 2022-04-10T01:37:15.909 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ seq 1 40 2022-04-10T01:37:15.910 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in `seq 1 40` 2022-04-10T01:37:15.910 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 1 -ne 1 2022-04-10T01:37:15.910 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2022-04-10T01:37:15.910 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep 'image health:' 2022-04-10T01:37:15.911 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cut '-d ' -f 3 2022-04-10T01:37:15.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: failed to query services: (13) Permission denied 2022-04-10T01:37:15.942 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ state=WARNING 2022-04-10T01:37:15.943 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test WARNING = ERROR 2022-04-10T01:37:15.943 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test WARNING = OK 2022-04-10T01:37:15.943 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in `seq 1 40` 2022-04-10T01:37:15.943 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 2 -ne 1 2022-04-10T01:37:15.944 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 30 2022-04-10T01:37:45.945 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2022-04-10T01:37:45.945 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep 'image health:' 2022-04-10T01:37:45.945 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cut '-d ' -f 3 2022-04-10T01:37:45.983 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: failed to query services: (13) Permission denied 2022-04-10T01:37:45.987 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ state=OK 2022-04-10T01:37:45.988 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test OK = ERROR 2022-04-10T01:37:45.988 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test OK = OK 2022-04-10T01:37:45.988 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:37:45.988 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_pool_images cluster1 mirror 50 2022-04-10T01:37:45.989 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:37:45.989 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:37:45.989 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_count=50 2022-04-10T01:37:45.989 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:37:45.990 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local count 2022-04-10T01:37:45.990 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local last_count=0 2022-04-10T01:37:45.990 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:37:45.990 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ seq 1 40 2022-04-10T01:37:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in `seq 1 40` 2022-04-10T01:37:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 1 -ne 1 2022-04-10T01:37:45.992 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2022-04-10T01:37:45.992 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep 'images: ' 2022-04-10T01:37:45.992 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cut '-d ' -f 2 2022-04-10T01:37:46.025 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: failed to query services: (13) Permission denied 2022-04-10T01:37:46.029 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ count=50 2022-04-10T01:37:46.029 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 50 = 50 2022-04-10T01:37:46.029 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:37:46.029 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_pool_healthy cluster1 mirror 2022-04-10T01:37:46.030 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:37:46.030 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:37:46.030 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:37:46.030 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state 2022-04-10T01:37:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ seq 1 40 2022-04-10T01:37:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in `seq 1 40` 2022-04-10T01:37:46.032 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 1 -ne 1 2022-04-10T01:37:46.032 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2022-04-10T01:37:46.032 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep 'image health:' 2022-04-10T01:37:46.032 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cut '-d ' -f 3 2022-04-10T01:37:46.064 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: failed to query services: (13) Permission denied 2022-04-10T01:37:46.068 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ state=OK 2022-04-10T01:37:46.068 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test OK = ERROR 2022-04-10T01:37:46.068 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test OK = OK 2022-04-10T01:37:46.069 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:37:46.069 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ testlog 'TEST: compare many images' 2022-04-10T01:37:46.069 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ tee -a /tmp/tmp.rgwz74WspX/rbd-mirror.test.log 2022-04-10T01:37:46.069 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ date '+%F %T' 2022-04-10T01:37:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo 2022-04-10 01:37:46 TEST: compare many images 2022-04-10T01:37:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:2022-04-10 01:37:46 TEST: compare many images 2022-04-10T01:37:46.072 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ seq 1 50 2022-04-10T01:37:46.072 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:37:46.072 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_1 2022-04-10T01:37:46.072 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror image_1 snap 2022-04-10T01:37:46.073 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:37:46.073 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:37:46.073 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_1 2022-04-10T01:37:46.073 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:37:46.074 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create image_1@snap --debug-rbd=20 --debug-journaler=20 2022-04-10T01:37:47.852 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror image_1 2022-04-10T01:37:47.852 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:37:47.853 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:37:47.853 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_1 2022-04-10T01:37:47.853 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:37:47.853 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:37:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:37:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:37:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror image_1 started 2022-04-10T01:37:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:37:47.855 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:37:47.855 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_1 2022-04-10T01:37:47.855 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:37:47.855 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:37:47.856 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:37:47.856 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:37:47.856 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:37:47.856 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:37:47.856 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:37:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:37:48.862 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror image_1 started 2022-04-10T01:37:48.863 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:37:48.863 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:37:48.863 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_1 2022-04-10T01:37:48.864 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:37:48.864 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:37:48.865 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:37:48.865 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:37:48.865 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:37:48.866 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:37:48.866 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:37:48.867 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_1 2022-04-10T01:37:48.867 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:37:48.867 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:37:48.868 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:37:48.868 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:37:48.868 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:37:48.869 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:37:48.869 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:37:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:37:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:37:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:37:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:37:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:37:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:37:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:37:48.873 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:37:48.873 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_1 2022-04-10T01:37:48.873 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:37:48.874 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:37:48.874 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:37:48.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:37:48.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:37:48.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:37:48.876 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:37:48.876 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:37:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:37:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:37:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:37:48.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:37:48.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:37:48.879 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:37:48.879 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:37:48.879 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:37:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:37:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:37:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:37:48.881 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:37:48.881 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:37:48.882 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:37:48.882 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:37:48.882 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:37:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:37:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:37:48.884 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:37:48.884 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:37:48.884 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:37:48.885 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:37:48.885 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:37:48.885 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:37:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:37:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:37:48.887 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:37:48.887 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:37:48.887 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:37:48.888 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:37:48.889 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:37:48.890 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:37:48.890 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:37:48.890 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_1 2022-04-10T01:37:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:no valid command found; 5 closest matches: 2022-04-10T01:37:48.991 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status mirror/image_11 2022-04-10T01:37:48.991 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status mirror/image_15 2022-04-10T01:37:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status mirror/image_17 2022-04-10T01:37:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status mirror/image_18 2022-04-10T01:37:48.993 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status mirror/image_19 2022-04-10T01:37:48.993 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:37:49.000 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:37:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_1 2022-04-10T01:37:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:1 2022-04-10T01:37:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:37:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:37:49.003 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:1 cluster instance 2022-04-10T01:37:49.003 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:1 2022-04-10T01:37:49.003 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:37:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:37:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:37:49.005 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=1 2022-04-10T01:37:49.005 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 1 = cluster1:1 ']' 2022-04-10T01:37:49.005 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:37:49.006 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:37:49.006 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=1 2022-04-10T01:37:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:37:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:37:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:37:49.008 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:37:49.008 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:37:49.009 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:1 cluster1 2022-04-10T01:37:49.009 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:1 2022-04-10T01:37:49.009 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:37:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:37:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2022-04-10T01:37:49.011 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:1 2022-04-10T01:37:49.011 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:37:49.011 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:37:49.012 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:37:49.012 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:37:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 1 = cluster1:1 ']' 2022-04-10T01:37:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:37:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:37:49.014 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=1 2022-04-10T01:37:49.014 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=1 2022-04-10T01:37:49.015 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:37:49.015 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:37:49.015 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:37:49.016 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:37:49.016 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2022-04-10T01:37:49.018 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:37:49.018 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:37:49.018 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:37:49.019 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_1 2022-04-10T01:37:49.124 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:37:49.124 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:37:49.125 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:37:49.125 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:37:49.126 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:37:49.137 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:37:49.138 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:37:49.138 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:37:49.139 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_1 2022-04-10T01:37:49.139 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:37:49.140 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:37:49.140 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:37:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_1 2022-04-10T01:37:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:37:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:37:49.142 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:37:49.142 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:37:49.143 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:37:49.143 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:37:49.143 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_1 2022-04-10T01:37:49.144 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:37:49.144 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:37:49.144 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:37:49.145 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_1 2022-04-10T01:37:49.145 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:37:49.146 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:37:49.146 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:37:49.147 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:37:49.147 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:37:49.147 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:37:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:37:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:37:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-image_1.status 2022-04-10T01:37:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-image_1.status 2022-04-10T01:37:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:37:49.150 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_1.status 2022-04-10T01:37:49.150 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-image_1.status 2022-04-10T01:37:49.151 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-image_1.status 2022-04-10T01:37:49.151 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:37:49.151 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_1.status 2022-04-10T01:37:49.152 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror image_1 2022-04-10T01:37:49.152 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:37:49.153 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:37:49.153 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_1 2022-04-10T01:37:49.153 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:37:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:37:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:37:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:37:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_1 2022-04-10T01:37:49.853 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 119 2022-04-10T01:37:49.863 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_1 /tmp/tmp.rgwz74WspX/cluster2-mirror-image_1.status 2022-04-10T01:37:49.864 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:37:49.864 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:37:49.864 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_1 2022-04-10T01:37:49.864 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_1.status 2022-04-10T01:37:49.865 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:37:49.865 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:37:49.865 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:37:49.865 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:37:49.866 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_1 --format xml 2022-04-10T01:37:49.866 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:37:49.905 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:37:49.907 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=119 2022-04-10T01:37:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:37:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:37:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:37:50.109 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_1.status 2022-04-10T01:37:50.109 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:37:50.110 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:37:50.110 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_1 2022-04-10T01:37:50.110 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_1.status 2022-04-10T01:37:50.110 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:37:50.111 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:37:50.111 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:37:50.111 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:37:50.111 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2022-04-10T01:37:50.112 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:37:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:37:50.157 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=68 2022-04-10T01:37:50.157 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 119 = 68 2022-04-10T01:37:50.157 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:37:50.158 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:37:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_1.status 2022-04-10T01:37:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:37:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:37:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_1 2022-04-10T01:37:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_1.status 2022-04-10T01:37:50.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:37:50.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:37:50.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:37:50.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:37:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2022-04-10T01:37:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:37:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:37:50.609 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=68 2022-04-10T01:37:50.609 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 119 = 68 2022-04-10T01:37:50.609 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:37:50.610 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:37:51.411 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_1.status 2022-04-10T01:37:51.411 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:37:51.411 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:37:51.412 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_1 2022-04-10T01:37:51.412 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_1.status 2022-04-10T01:37:51.412 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:37:51.412 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:37:51.413 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:37:51.413 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:37:51.413 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2022-04-10T01:37:51.413 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:37:51.456 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:37:51.456 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:37:51.457 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:37:51.458 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:37:51.458 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:37:51.459 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:37:51.459 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 119 = '' 2022-04-10T01:37:51.459 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:37:51.459 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1.6 2022-04-10T01:37:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_1.status 2022-04-10T01:37:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:37:53.061 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:37:53.061 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_1 2022-04-10T01:37:53.061 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_1.status 2022-04-10T01:37:53.061 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:37:53.061 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:37:53.062 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:37:53.062 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:37:53.062 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2022-04-10T01:37:53.062 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:37:53.108 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:37:53.111 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=119 2022-04-10T01:37:53.111 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 119 = 119 2022-04-10T01:37:53.111 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:37:53.112 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror image_1 snap 2022-04-10T01:37:53.112 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:37:53.112 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:37:53.112 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_1 2022-04-10T01:37:53.113 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:37:53.113 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:37:53.113 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:37:53.113 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:37:53.114 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:37:53.114 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:37:53.114 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:37:53.114 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:37:54.111 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/image_1@snap 2022-04-10T01:37:54.155 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_1': 2022-04-10T01:37:54.156 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:37:54.156 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:37:54.156 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 2 2022-04-10T01:37:54.157 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 10e4d8f9444e 2022-04-10T01:37:54.157 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.10e4d8f9444e 2022-04-10T01:37:54.157 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:37:54.157 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:37:54.157 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:37:54.158 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:37:54.158 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:28:03 2022 2022-04-10T01:37:54.158 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:28:03 2022 2022-04-10T01:37:54.158 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:28:03 2022 2022-04-10T01:37:54.159 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:37:54.159 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:37:54.159 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:37:54.159 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: acfa4173-7a18-4ae9-975a-94af9fdf80e4 2022-04-10T01:37:54.160 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:37:54.161 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:37:54.161 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror image_1 snap 2022-04-10T01:37:54.161 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:37:54.162 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_1 2022-04-10T01:37:54.162 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:37:54.162 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:37:54.162 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_1.export 2022-04-10T01:37:54.163 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_1.export 2022-04-10T01:37:54.163 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_1.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_1.export 2022-04-10T01:37:54.163 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export image_1@snap /tmp/tmp.rgwz74WspX/cluster2-mirror-image_1.export 2022-04-10T01:37:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:37:54.342 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export image_1@snap /tmp/tmp.rgwz74WspX/cluster1-mirror-image_1.export 2022-04-10T01:37:54.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:37:54.585 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-image_1.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_1.export 2022-04-10T01:37:54.650 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_1.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_1.export 2022-04-10T01:37:54.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:37:54.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:37:54.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_2 2022-04-10T01:37:54.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror image_2 snap 2022-04-10T01:37:54.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:37:54.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:37:54.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_2 2022-04-10T01:37:54.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:37:54.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create image_2@snap --debug-rbd=20 --debug-journaler=20 2022-04-10T01:37:56.474 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror image_2 2022-04-10T01:37:56.474 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:37:56.474 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:37:56.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_2 2022-04-10T01:37:56.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:37:56.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:37:56.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:37:56.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:37:56.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror image_2 started 2022-04-10T01:37:56.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:37:56.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:37:56.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_2 2022-04-10T01:37:56.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:37:56.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:37:56.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:37:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:37:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:37:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:37:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:37:56.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:37:57.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror image_2 started 2022-04-10T01:37:57.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:37:57.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:37:57.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_2 2022-04-10T01:37:57.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:37:57.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:37:57.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:37:57.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:37:57.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:37:57.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:37:57.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:37:57.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_2 2022-04-10T01:37:57.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:37:57.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:37:57.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:37:57.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:37:57.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:37:57.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:37:57.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:37:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:37:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:37:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:37:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:37:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:37:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:37:57.486 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:37:57.486 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:37:57.487 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2022-04-10T01:37:57.487 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:37:57.487 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:37:57.487 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:37:57.487 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:37:57.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:37:57.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:37:57.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:37:57.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:37:57.489 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:37:57.489 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:37:57.489 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:37:57.489 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:37:57.489 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:37:57.490 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:37:57.490 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:37:57.490 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:37:57.490 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:37:57.491 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:37:57.491 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:37:57.491 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:37:57.491 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:37:57.491 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:37:57.492 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:37:57.492 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:37:57.492 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:37:57.492 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:37:57.492 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:37:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:37:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:37:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:37:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:37:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:37:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:37:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:37:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:37:57.495 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:37:57.495 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:37:57.495 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:37:57.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:37:57.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:37:57.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:37:57.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2022-04-10T01:37:57.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:no valid command found; 3 closest matches: 2022-04-10T01:37:57.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status mirror/image_21 2022-04-10T01:37:57.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status mirror/image_22 2022-04-10T01:37:57.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status mirror/image_23 2022-04-10T01:37:57.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:37:57.617 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:37:57.617 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_2 2022-04-10T01:37:57.617 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:1 2022-04-10T01:37:57.617 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:37:57.618 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:37:57.618 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:1 cluster instance 2022-04-10T01:37:57.618 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:1 2022-04-10T01:37:57.618 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:37:57.618 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:37:57.619 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:37:57.619 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=1 2022-04-10T01:37:57.619 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 1 = cluster1:1 ']' 2022-04-10T01:37:57.619 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:37:57.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:37:57.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=1 2022-04-10T01:37:57.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:37:57.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:37:57.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:37:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:37:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:37:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:1 cluster1 2022-04-10T01:37:57.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:1 2022-04-10T01:37:57.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:37:57.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:37:57.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2022-04-10T01:37:57.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:1 2022-04-10T01:37:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:37:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:37:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:37:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:37:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 1 = cluster1:1 ']' 2022-04-10T01:37:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:37:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:37:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=1 2022-04-10T01:37:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=1 2022-04-10T01:37:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:37:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:37:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:37:57.626 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:37:57.626 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2022-04-10T01:37:57.630 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:37:57.631 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:37:57.631 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:37:57.631 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_2 2022-04-10T01:37:57.735 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:no valid command found; 2 closest matches: 2022-04-10T01:37:57.735 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status mirror/image_27 2022-04-10T01:37:57.736 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status mirror/image_28 2022-04-10T01:37:57.736 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:37:57.740 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:37:57.740 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_2 2022-04-10T01:37:57.740 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:2 2022-04-10T01:37:57.740 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:37:57.741 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:37:57.741 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:2 cluster instance 2022-04-10T01:37:57.741 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:2 2022-04-10T01:37:57.741 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:37:57.742 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:37:57.742 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:37:57.742 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=2 2022-04-10T01:37:57.742 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 2 = cluster1:2 ']' 2022-04-10T01:37:57.743 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:37:57.743 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:37:57.743 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=2 2022-04-10T01:37:57.743 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:37:57.744 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:37:57.744 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:37:57.744 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:37:57.744 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:37:57.745 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:2 cluster1 2022-04-10T01:37:57.745 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:2 2022-04-10T01:37:57.745 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:37:57.745 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:37:57.745 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2022-04-10T01:37:57.746 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:2 2022-04-10T01:37:57.746 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:37:57.746 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:37:57.746 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:37:57.747 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:37:57.747 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 2 = cluster1:2 ']' 2022-04-10T01:37:57.747 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:37:57.748 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:37:57.748 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=2 2022-04-10T01:37:57.748 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=2 2022-04-10T01:37:57.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:37:57.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:37:57.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:37:57.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:37:57.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2022-04-10T01:37:57.754 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:37:57.754 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:37:57.754 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:37:57.754 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_2 2022-04-10T01:37:57.869 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:37:57.870 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:37:57.870 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:37:57.870 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:37:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:37:57.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:37:57.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:37:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:37:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_2 2022-04-10T01:37:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:37:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:37:57.877 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:37:57.877 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_2 2022-04-10T01:37:57.877 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:37:57.877 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:37:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:37:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:37:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:37:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:37:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_2 2022-04-10T01:37:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:37:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:37:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:37:57.880 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_2 2022-04-10T01:37:57.880 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:37:57.880 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:37:57.880 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:37:57.880 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:37:57.881 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:37:57.881 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:37:57.881 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:37:57.881 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:37:57.882 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-image_2.status 2022-04-10T01:37:57.882 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-image_2.status 2022-04-10T01:37:57.882 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:37:57.882 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_2.status 2022-04-10T01:37:57.883 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-image_2.status 2022-04-10T01:37:57.883 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-image_2.status 2022-04-10T01:37:57.883 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:37:57.883 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_2.status 2022-04-10T01:37:57.883 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror image_2 2022-04-10T01:37:57.884 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:37:57.884 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:37:57.884 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_2 2022-04-10T01:37:57.884 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:37:57.884 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:37:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:37:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:37:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_2 2022-04-10T01:37:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 122 2022-04-10T01:37:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_2 /tmp/tmp.rgwz74WspX/cluster2-mirror-image_2.status 2022-04-10T01:37:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:37:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:37:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_2 2022-04-10T01:37:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_2.status 2022-04-10T01:37:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:37:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:37:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:37:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:37:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_2 --format xml 2022-04-10T01:37:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:37:58.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:37:58.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=122 2022-04-10T01:37:58.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:37:58.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:37:58.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:37:58.741 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_2.status 2022-04-10T01:37:58.741 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:37:58.741 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:37:58.741 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_2 2022-04-10T01:37:58.742 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_2.status 2022-04-10T01:37:58.742 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:37:58.742 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:37:58.742 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:37:58.743 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:37:58.743 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2022-04-10T01:37:58.743 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:37:58.785 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:37:58.786 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:37:58.787 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:37:58.787 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:37:58.787 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:37:58.788 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:37:58.788 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 122 = '' 2022-04-10T01:37:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:37:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:37:59.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_2.status 2022-04-10T01:37:59.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:37:59.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:37:59.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_2 2022-04-10T01:37:59.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_2.status 2022-04-10T01:37:59.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:37:59.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:37:59.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:37:59.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:37:59.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2022-04-10T01:37:59.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:37:59.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:37:59.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:37:59.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:37:59.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:37:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:37:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:37:59.242 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 122 = '' 2022-04-10T01:37:59.242 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:37:59.242 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:38:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_2.status 2022-04-10T01:38:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_2 2022-04-10T01:38:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_2.status 2022-04-10T01:38:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:38:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2022-04-10T01:38:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:38:00.084 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:38:00.087 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=122 2022-04-10T01:38:00.087 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 122 = 122 2022-04-10T01:38:00.087 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:00.088 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror image_2 snap 2022-04-10T01:38:00.088 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:00.088 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:00.088 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_2 2022-04-10T01:38:00.089 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:38:00.089 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:38:00.089 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:38:00.089 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:00.089 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:38:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:38:01.087 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/image_2@snap 2022-04-10T01:38:01.132 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_2': 2022-04-10T01:38:01.132 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:38:01.132 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:38:01.132 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 2 2022-04-10T01:38:01.133 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 113390f26957 2022-04-10T01:38:01.133 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.113390f26957 2022-04-10T01:38:01.133 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:38:01.133 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:38:01.134 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:38:01.134 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:38:01.134 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:28:11 2022 2022-04-10T01:38:01.134 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:28:11 2022 2022-04-10T01:38:01.135 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:28:11 2022 2022-04-10T01:38:01.135 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:38:01.135 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:38:01.135 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:38:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: a0c16b90-9fb2-4d3c-834f-4b590db7c01d 2022-04-10T01:38:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:38:01.138 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:01.138 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror image_2 snap 2022-04-10T01:38:01.138 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:01.138 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_2 2022-04-10T01:38:01.139 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:38:01.139 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:38:01.139 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_2.export 2022-04-10T01:38:01.139 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_2.export 2022-04-10T01:38:01.140 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_2.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_2.export 2022-04-10T01:38:01.140 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export image_2@snap /tmp/tmp.rgwz74WspX/cluster2-mirror-image_2.export 2022-04-10T01:38:01.313 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:38:01.316 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export image_2@snap /tmp/tmp.rgwz74WspX/cluster1-mirror-image_2.export 2022-04-10T01:38:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:38:01.557 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-image_2.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_2.export 2022-04-10T01:38:01.621 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_2.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_2.export 2022-04-10T01:38:01.659 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:01.659 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:38:01.660 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_3 2022-04-10T01:38:01.660 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror image_3 snap 2022-04-10T01:38:01.660 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:38:01.660 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:01.661 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_3 2022-04-10T01:38:01.661 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:38:01.661 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create image_3@snap --debug-rbd=20 --debug-journaler=20 2022-04-10T01:38:03.493 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror image_3 2022-04-10T01:38:03.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:03.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:03.495 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_3 2022-04-10T01:38:03.495 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:38:03.495 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:03.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:03.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:03.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror image_3 started 2022-04-10T01:38:03.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:03.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_3 2022-04-10T01:38:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:38:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:38:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:38:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:38:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:38:04.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror image_3 started 2022-04-10T01:38:04.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:04.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:04.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_3 2022-04-10T01:38:04.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:38:04.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:38:04.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:38:04.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:38:04.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:04.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_3 2022-04-10T01:38:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:38:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:38:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:38:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:38:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:38:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:38:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:38:04.501 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:38:04.501 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:04.501 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:38:04.501 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:38:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:38:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:38:04.506 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:38:04.506 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:38:04.506 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_3 2022-04-10T01:38:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:38:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:38:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:38:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:38:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:38:04.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:38:04.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:38:04.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:38:04.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:38:04.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:38:04.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:38:04.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:04.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:38:04.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:38:04.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:38:04.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:38:04.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:04.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:38:04.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:38:04.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:38:04.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:04.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:38:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:38:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:38:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:38:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:38:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:04.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:38:04.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:38:04.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:38:04.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:38:04.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:38:04.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:38:04.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:38:04.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:04.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:04.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:38:04.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:38:04.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:38:04.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:38:04.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:38:04.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_3 2022-04-10T01:38:04.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:38:04.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:38:04.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:38:04.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:38:04.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:38:04.647 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:38:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:38:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_3 2022-04-10T01:38:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:38:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:38:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_3 2022-04-10T01:38:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:38:04.650 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:38:04.650 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:04.650 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:04.650 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:38:04.650 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:38:04.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_3 2022-04-10T01:38:04.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:38:04.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:38:04.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:04.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_3 2022-04-10T01:38:04.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:38:04.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:04.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:04.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:04.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:38:04.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:38:04.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:04.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:04.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-image_3.status 2022-04-10T01:38:04.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-image_3.status 2022-04-10T01:38:04.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:38:04.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_3.status 2022-04-10T01:38:04.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-image_3.status 2022-04-10T01:38:04.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-image_3.status 2022-04-10T01:38:04.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:38:04.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_3.status 2022-04-10T01:38:04.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror image_3 2022-04-10T01:38:04.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:38:04.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:04.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_3 2022-04-10T01:38:04.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:38:04.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:38:04.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:04.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:04.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_3 2022-04-10T01:38:05.501 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 125 2022-04-10T01:38:05.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_3 /tmp/tmp.rgwz74WspX/cluster2-mirror-image_3.status 2022-04-10T01:38:05.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:38:05.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:05.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_3 2022-04-10T01:38:05.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_3.status 2022-04-10T01:38:05.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:38:05.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:38:05.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:05.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:05.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_3 --format xml 2022-04-10T01:38:05.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:38:05.556 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:38:05.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=125 2022-04-10T01:38:05.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:38:05.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:38:05.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:38:05.760 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_3.status 2022-04-10T01:38:05.761 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:05.761 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:05.761 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_3 2022-04-10T01:38:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_3.status 2022-04-10T01:38:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:38:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:05.763 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:05.763 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2022-04-10T01:38:05.763 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:38:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:38:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:38:05.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:38:05.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:38:05.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:38:05.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:38:05.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 125 = '' 2022-04-10T01:38:05.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:38:05.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:38:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_3.status 2022-04-10T01:38:06.209 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:06.209 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:06.209 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_3 2022-04-10T01:38:06.210 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_3.status 2022-04-10T01:38:06.210 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:38:06.210 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2022-04-10T01:38:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:38:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:38:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:38:06.259 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:38:06.259 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:38:06.260 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:38:06.260 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:38:06.261 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 125 = '' 2022-04-10T01:38:06.261 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:38:06.261 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:38:07.062 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_3.status 2022-04-10T01:38:07.062 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:07.063 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:07.063 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_3 2022-04-10T01:38:07.063 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_3.status 2022-04-10T01:38:07.063 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:38:07.064 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:07.064 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:07.064 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:07.064 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2022-04-10T01:38:07.064 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:38:07.104 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:38:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=125 2022-04-10T01:38:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 125 = 125 2022-04-10T01:38:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:07.108 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror image_3 snap 2022-04-10T01:38:07.108 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:07.108 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:07.108 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_3 2022-04-10T01:38:07.108 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:38:07.109 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:38:07.109 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:38:07.109 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:07.109 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:07.110 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:07.110 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:38:07.110 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:38:08.107 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/image_3@snap 2022-04-10T01:38:08.152 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_3': 2022-04-10T01:38:08.152 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:38:08.152 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:38:08.153 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 2 2022-04-10T01:38:08.153 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 118d4a93dcfc 2022-04-10T01:38:08.153 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.118d4a93dcfc 2022-04-10T01:38:08.153 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:38:08.153 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:38:08.154 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:38:08.154 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:38:08.154 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:28:22 2022 2022-04-10T01:38:08.154 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:28:22 2022 2022-04-10T01:38:08.155 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:28:22 2022 2022-04-10T01:38:08.155 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:38:08.155 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:38:08.155 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:38:08.155 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 6d595122-4619-4d21-a653-5b0cde552987 2022-04-10T01:38:08.156 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:38:08.157 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:08.157 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror image_3 snap 2022-04-10T01:38:08.158 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:08.158 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_3 2022-04-10T01:38:08.158 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:38:08.158 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:38:08.159 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_3.export 2022-04-10T01:38:08.159 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_3.export 2022-04-10T01:38:08.159 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_3.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_3.export 2022-04-10T01:38:08.159 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export image_3@snap /tmp/tmp.rgwz74WspX/cluster2-mirror-image_3.export 2022-04-10T01:38:08.344 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:38:08.348 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export image_3@snap /tmp/tmp.rgwz74WspX/cluster1-mirror-image_3.export 2022-04-10T01:38:08.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:38:08.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-image_3.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_3.export 2022-04-10T01:38:08.662 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_3.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_3.export 2022-04-10T01:38:08.700 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:08.700 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:38:08.700 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_4 2022-04-10T01:38:08.701 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror image_4 snap 2022-04-10T01:38:08.701 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:38:08.701 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:08.701 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_4 2022-04-10T01:38:08.702 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:38:08.702 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create image_4@snap --debug-rbd=20 --debug-journaler=20 2022-04-10T01:38:10.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror image_4 2022-04-10T01:38:10.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:10.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:10.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_4 2022-04-10T01:38:10.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:38:10.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:10.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:10.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:10.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror image_4 started 2022-04-10T01:38:10.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:10.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:10.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_4 2022-04-10T01:38:10.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:38:10.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:38:10.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:38:10.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:10.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:10.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:10.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:38:10.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:38:11.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror image_4 started 2022-04-10T01:38:11.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:11.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:11.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_4 2022-04-10T01:38:11.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:38:11.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:38:11.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:38:11.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:38:11.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:11.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:11.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:11.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_4 2022-04-10T01:38:11.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:38:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:38:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:38:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:38:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:38:11.483 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:38:11.483 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:38:11.483 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:38:11.483 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:11.483 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:38:11.484 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:38:11.484 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:38:11.487 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:38:11.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:38:11.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:38:11.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2022-04-10T01:38:11.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:38:11.489 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:38:11.489 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:38:11.489 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:38:11.489 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:38:11.489 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:38:11.490 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:38:11.490 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:38:11.490 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:38:11.490 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:38:11.491 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:38:11.491 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:11.491 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:38:11.491 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:38:11.491 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:38:11.492 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:38:11.492 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:11.492 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:38:11.492 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:38:11.492 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:38:11.493 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:11.493 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:38:11.493 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:38:11.493 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:38:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:38:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:38:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:38:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:38:11.495 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:38:11.495 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:38:11.495 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:38:11.495 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:38:11.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:38:11.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:11.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:11.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:38:11.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:38:11.501 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:38:11.501 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:38:11.501 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:38:11.502 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2022-04-10T01:38:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:no valid command found; 1 closest matches: 2022-04-10T01:38:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status mirror/image_43 2022-04-10T01:38:11.616 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:38:11.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:38:11.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_4 2022-04-10T01:38:11.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:1 2022-04-10T01:38:11.621 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:38:11.621 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:38:11.621 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:1 cluster instance 2022-04-10T01:38:11.621 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:1 2022-04-10T01:38:11.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:38:11.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:38:11.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:38:11.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=1 2022-04-10T01:38:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 1 = cluster1:1 ']' 2022-04-10T01:38:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:38:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=1 2022-04-10T01:38:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:38:11.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:38:11.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:38:11.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:11.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:38:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:1 cluster1 2022-04-10T01:38:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:1 2022-04-10T01:38:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:38:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2022-04-10T01:38:11.626 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:1 2022-04-10T01:38:11.626 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:38:11.626 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:38:11.626 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:38:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 1 = cluster1:1 ']' 2022-04-10T01:38:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:38:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:38:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=1 2022-04-10T01:38:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=1 2022-04-10T01:38:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:38:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:38:11.629 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2022-04-10T01:38:11.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:38:11.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:38:11.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:38:11.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_4 2022-04-10T01:38:11.747 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:38:11.748 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:38:11.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:38:11.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:38:11.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:38:11.752 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:38:11.753 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:38:11.753 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:11.753 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_4 2022-04-10T01:38:11.753 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:38:11.754 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:38:11.754 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:11.754 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_4 2022-04-10T01:38:11.754 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:38:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:38:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:38:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:38:11.756 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_4 2022-04-10T01:38:11.756 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:38:11.756 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:38:11.756 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:11.756 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_4 2022-04-10T01:38:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:38:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:38:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:38:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-image_4.status 2022-04-10T01:38:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-image_4.status 2022-04-10T01:38:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:38:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_4.status 2022-04-10T01:38:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-image_4.status 2022-04-10T01:38:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-image_4.status 2022-04-10T01:38:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:38:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_4.status 2022-04-10T01:38:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror image_4 2022-04-10T01:38:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:38:11.762 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:11.762 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_4 2022-04-10T01:38:11.762 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:38:11.762 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:38:11.762 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_4 2022-04-10T01:38:12.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 128 2022-04-10T01:38:12.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_4 /tmp/tmp.rgwz74WspX/cluster2-mirror-image_4.status 2022-04-10T01:38:12.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:38:12.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:12.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_4 2022-04-10T01:38:12.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_4.status 2022-04-10T01:38:12.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:38:12.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:38:12.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:12.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:12.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_4 --format xml 2022-04-10T01:38:12.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:38:12.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:38:12.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=128 2022-04-10T01:38:12.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:38:12.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:38:12.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:38:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_4.status 2022-04-10T01:38:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_4 2022-04-10T01:38:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_4.status 2022-04-10T01:38:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:38:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2022-04-10T01:38:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:38:12.813 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:38:12.813 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:38:12.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:38:12.816 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:38:12.816 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:38:12.816 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:38:12.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 128 = '' 2022-04-10T01:38:12.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:38:12.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:38:13.217 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_4.status 2022-04-10T01:38:13.218 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:13.218 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:13.218 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_4 2022-04-10T01:38:13.218 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_4.status 2022-04-10T01:38:13.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:38:13.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:13.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:13.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:13.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2022-04-10T01:38:13.220 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:38:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:38:13.266 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:38:13.267 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:38:13.267 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:38:13.267 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:38:13.268 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:38:13.268 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 128 = '' 2022-04-10T01:38:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:38:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:38:14.070 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_4.status 2022-04-10T01:38:14.070 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:14.070 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_4 2022-04-10T01:38:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_4.status 2022-04-10T01:38:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:38:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:14.072 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:14.072 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2022-04-10T01:38:14.072 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:38:14.146 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:38:14.149 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=128 2022-04-10T01:38:14.149 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 128 = 128 2022-04-10T01:38:14.149 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:14.149 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror image_4 snap 2022-04-10T01:38:14.150 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:14.150 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:14.150 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_4 2022-04-10T01:38:14.150 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:38:14.150 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:38:14.151 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:38:14.151 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:14.151 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:14.151 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:14.151 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:38:14.152 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:38:14.160 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.45696608996703536) > weight (0.25) 2022-04-10T01:38:14.161 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2022-04-10T01:38:14.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2022-04-10T01:38:14.162 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2022-04-10T01:38:14.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi043.stderr:2022-04-10T01:38:14.160+0000 7fb8606c6700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 48888) UID: 1000 2022-04-10T01:38:14.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi043.stderr:daemon-helper: command failed with exit status 1 2022-04-10T01:38:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/image_4@snap 2022-04-10T01:38:15.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_4': 2022-04-10T01:38:15.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:38:15.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:38:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 2 2022-04-10T01:38:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 128782dbce0d 2022-04-10T01:38:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.128782dbce0d 2022-04-10T01:38:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:38:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:38:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:38:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:38:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:28:36 2022 2022-04-10T01:38:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:28:36 2022 2022-04-10T01:38:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:28:36 2022 2022-04-10T01:38:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:38:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:38:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:38:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: d4a4b534-85cd-4706-8f4e-137718b81a97 2022-04-10T01:38:15.194 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:38:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror image_4 snap 2022-04-10T01:38:15.196 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:15.196 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_4 2022-04-10T01:38:15.196 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:38:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:38:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_4.export 2022-04-10T01:38:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_4.export 2022-04-10T01:38:15.198 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_4.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_4.export 2022-04-10T01:38:15.198 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export image_4@snap /tmp/tmp.rgwz74WspX/cluster2-mirror-image_4.export 2022-04-10T01:38:15.378 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:38:15.382 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export image_4@snap /tmp/tmp.rgwz74WspX/cluster1-mirror-image_4.export 2022-04-10T01:38:15.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:38:15.648 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-image_4.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_4.export 2022-04-10T01:38:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_4.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_4.export 2022-04-10T01:38:15.752 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:15.753 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:38:15.753 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_5 2022-04-10T01:38:15.753 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror image_5 snap 2022-04-10T01:38:15.754 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:38:15.754 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:15.754 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_5 2022-04-10T01:38:15.754 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:38:15.755 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create image_5@snap --debug-rbd=20 --debug-journaler=20 2022-04-10T01:38:17.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror image_5 2022-04-10T01:38:17.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:17.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:17.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_5 2022-04-10T01:38:17.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:38:17.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:17.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror image_5 started 2022-04-10T01:38:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_5 2022-04-10T01:38:17.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:38:17.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:38:17.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:38:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:38:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:38:18.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror image_5 started 2022-04-10T01:38:18.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:18.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:18.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_5 2022-04-10T01:38:18.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:38:18.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:38:18.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:38:18.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:38:18.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:18.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:18.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:18.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_5 2022-04-10T01:38:18.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:38:18.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:38:18.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:38:18.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:38:18.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:38:18.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:38:18.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:38:18.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:38:18.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:18.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:38:18.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:38:18.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:38:18.528 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:38:18.528 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:38:18.532 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:38:18.532 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2022-04-10T01:38:18.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:38:18.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:38:18.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:38:18.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:38:18.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:38:18.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:38:18.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:38:18.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:38:18.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:38:18.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:38:18.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:38:18.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:18.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:38:18.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:38:18.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:38:18.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:38:18.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:18.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:38:18.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:38:18.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:38:18.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:18.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:38:18.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:38:18.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:38:18.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:38:18.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:38:18.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:18.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:38:18.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:38:18.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:38:18.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:38:18.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:38:18.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:38:18.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:38:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:38:18.542 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:38:18.544 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:38:18.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:38:18.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:38:18.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2022-04-10T01:38:18.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2022-04-10T01:38:18.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:38:18.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:38:18.662 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:38:18.662 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_5 2022-04-10T01:38:18.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:1 2022-04-10T01:38:18.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:38:18.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:38:18.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:1 cluster instance 2022-04-10T01:38:18.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:1 2022-04-10T01:38:18.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:38:18.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:38:18.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:38:18.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=1 2022-04-10T01:38:18.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 1 = cluster1:1 ']' 2022-04-10T01:38:18.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:38:18.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:18.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=1 2022-04-10T01:38:18.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:38:18.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:38:18.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:38:18.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:18.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:38:18.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:1 cluster1 2022-04-10T01:38:18.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:1 2022-04-10T01:38:18.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:18.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:38:18.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2022-04-10T01:38:18.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:1 2022-04-10T01:38:18.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:38:18.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:38:18.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:18.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:38:18.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 1 = cluster1:1 ']' 2022-04-10T01:38:18.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:38:18.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:38:18.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=1 2022-04-10T01:38:18.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=1 2022-04-10T01:38:18.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:38:18.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:18.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:18.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:38:18.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2022-04-10T01:38:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:38:18.677 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:38:18.677 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:38:18.677 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_5 2022-04-10T01:38:18.791 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:38:18.792 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:38:18.792 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:38:18.792 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:38:18.793 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:38:18.797 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:38:18.797 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:38:18.798 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:18.798 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_5 2022-04-10T01:38:18.798 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:38:18.798 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:38:18.799 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:18.799 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_5 2022-04-10T01:38:18.799 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:38:18.799 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:38:18.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:18.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:18.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:38:18.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:38:18.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_5 2022-04-10T01:38:18.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:38:18.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:38:18.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:18.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_5 2022-04-10T01:38:18.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:38:18.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:18.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:18.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:18.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:38:18.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:38:18.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:18.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:18.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-image_5.status 2022-04-10T01:38:18.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-image_5.status 2022-04-10T01:38:18.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:38:18.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_5.status 2022-04-10T01:38:18.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-image_5.status 2022-04-10T01:38:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-image_5.status 2022-04-10T01:38:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:38:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_5.status 2022-04-10T01:38:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror image_5 2022-04-10T01:38:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:38:18.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:18.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_5 2022-04-10T01:38:18.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:38:18.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:38:18.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:18.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:18.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_5 2022-04-10T01:38:19.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 131 2022-04-10T01:38:19.532 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_5 /tmp/tmp.rgwz74WspX/cluster2-mirror-image_5.status 2022-04-10T01:38:19.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:38:19.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:19.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_5 2022-04-10T01:38:19.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_5.status 2022-04-10T01:38:19.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:38:19.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:38:19.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:19.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:19.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_5 --format xml 2022-04-10T01:38:19.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:38:19.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:38:19.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=131 2022-04-10T01:38:19.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:38:19.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:38:19.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:38:19.778 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_5.status 2022-04-10T01:38:19.778 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:19.778 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_5 2022-04-10T01:38:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_5.status 2022-04-10T01:38:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:38:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:19.780 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:19.780 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:19.780 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2022-04-10T01:38:19.780 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:38:19.823 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:38:19.823 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:38:19.824 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:38:19.824 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:38:19.825 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:38:19.826 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:38:19.826 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 131 = '' 2022-04-10T01:38:19.826 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:38:19.826 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:38:20.226 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_5.status 2022-04-10T01:38:20.227 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:20.227 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:20.227 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_5 2022-04-10T01:38:20.227 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_5.status 2022-04-10T01:38:20.228 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:38:20.228 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:20.228 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:20.228 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:20.228 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2022-04-10T01:38:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:38:20.274 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:38:20.275 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:38:20.277 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:38:20.277 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:38:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:38:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:38:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 131 = '' 2022-04-10T01:38:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:38:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:38:21.078 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_5.status 2022-04-10T01:38:21.079 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:21.079 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:21.079 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_5 2022-04-10T01:38:21.079 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_5.status 2022-04-10T01:38:21.080 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:38:21.080 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:21.080 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:21.080 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:21.081 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2022-04-10T01:38:21.081 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:38:21.119 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:38:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=131 2022-04-10T01:38:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 131 = 131 2022-04-10T01:38:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror image_5 snap 2022-04-10T01:38:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_5 2022-04-10T01:38:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:38:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:38:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:38:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:21.126 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:38:21.126 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:38:22.123 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/image_5@snap 2022-04-10T01:38:22.163 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2022-04-10T01:38:22.163 INFO:teuthology.orchestra.run:waiting for 600 2022-04-10T01:38:22.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-04-10T01:38:22.164 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2022-04-10T01:38:22.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2022-04-10T01:38:22.165 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2022-04-10T01:38:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_5': 2022-04-10T01:38:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:38:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:38:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 2 2022-04-10T01:38:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 10e489288849 2022-04-10T01:38:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.10e489288849 2022-04-10T01:38:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:38:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:38:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:38:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:38:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:28:50 2022 2022-04-10T01:38:22.169 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:28:50 2022 2022-04-10T01:38:22.169 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:28:50 2022 2022-04-10T01:38:22.169 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:38:22.169 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:38:22.170 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:38:22.170 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 04a20513-9ad9-463b-ad2e-0f637bb8e1de 2022-04-10T01:38:22.170 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:38:22.172 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:22.172 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror image_5 snap 2022-04-10T01:38:22.172 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:22.173 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_5 2022-04-10T01:38:22.173 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:38:22.173 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:38:22.173 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_5.export 2022-04-10T01:38:22.173 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_5.export 2022-04-10T01:38:22.174 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_5.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_5.export 2022-04-10T01:38:22.174 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export image_5@snap /tmp/tmp.rgwz74WspX/cluster2-mirror-image_5.export 2022-04-10T01:38:22.175 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2022-04-10T01:38:22.175 INFO:tasks.rbd_mirror_thrash:waiting for 108 secs before thrashing 2022-04-10T01:38:22.360 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:38:22.364 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export image_5@snap /tmp/tmp.rgwz74WspX/cluster1-mirror-image_5.export 2022-04-10T01:38:22.615 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:38:22.618 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-image_5.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_5.export 2022-04-10T01:38:22.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_5.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_5.export 2022-04-10T01:38:22.720 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:22.720 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:38:22.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_6 2022-04-10T01:38:22.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror image_6 snap 2022-04-10T01:38:22.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:38:22.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:22.722 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_6 2022-04-10T01:38:22.722 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:38:22.722 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create image_6@snap --debug-rbd=20 --debug-journaler=20 2022-04-10T01:38:24.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror image_6 2022-04-10T01:38:24.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:24.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:24.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_6 2022-04-10T01:38:24.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:38:24.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:24.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:24.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:24.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror image_6 started 2022-04-10T01:38:24.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:24.483 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:24.483 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_6 2022-04-10T01:38:24.483 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:38:24.483 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:38:24.483 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:38:24.484 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:24.484 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:24.484 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:24.484 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:38:24.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:38:25.483 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror image_6 started 2022-04-10T01:38:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_6 2022-04-10T01:38:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:38:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:38:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:38:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:38:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_6 2022-04-10T01:38:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:38:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:38:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:38:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:38:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:38:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:38:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:38:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:38:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:38:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:38:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:38:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:38:25.490 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:38:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:38:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_6 2022-04-10T01:38:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:38:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:38:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:38:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:38:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:38:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:38:25.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:38:25.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:38:25.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:38:25.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:38:25.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:38:25.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:25.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:38:25.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:38:25.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:38:25.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:38:25.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:25.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:38:25.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:38:25.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:38:25.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:25.500 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:38:25.500 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:38:25.500 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:38:25.500 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:38:25.501 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:38:25.501 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:25.501 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:38:25.501 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:38:25.502 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:38:25.502 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:38:25.502 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:38:25.502 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:38:25.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:38:25.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:25.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:25.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:38:25.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:38:25.505 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:38:25.505 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:38:25.505 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:38:25.506 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_6 2022-04-10T01:38:25.630 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:38:25.630 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:38:25.631 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:38:25.631 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:38:25.632 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:38:25.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:38:25.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:38:25.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:25.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_6 2022-04-10T01:38:25.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:38:25.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:38:25.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:25.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_6 2022-04-10T01:38:25.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:38:25.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:38:25.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:25.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:25.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:38:25.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:38:25.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_6 2022-04-10T01:38:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:38:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:38:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_6 2022-04-10T01:38:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:38:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:38:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:38:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-image_6.status 2022-04-10T01:38:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-image_6.status 2022-04-10T01:38:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:38:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_6.status 2022-04-10T01:38:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-image_6.status 2022-04-10T01:38:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-image_6.status 2022-04-10T01:38:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:38:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_6.status 2022-04-10T01:38:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror image_6 2022-04-10T01:38:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:38:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_6 2022-04-10T01:38:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:38:25.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:38:25.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:25.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:25.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_6 2022-04-10T01:38:26.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 134 2022-04-10T01:38:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_6 /tmp/tmp.rgwz74WspX/cluster2-mirror-image_6.status 2022-04-10T01:38:26.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:38:26.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:26.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_6 2022-04-10T01:38:26.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_6.status 2022-04-10T01:38:26.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:38:26.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:38:26.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:26.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:26.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_6 --format xml 2022-04-10T01:38:26.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:38:26.588 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:38:26.590 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=134 2022-04-10T01:38:26.591 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:38:26.591 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:38:26.591 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:38:26.792 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_6.status 2022-04-10T01:38:26.793 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:26.793 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:26.793 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_6 2022-04-10T01:38:26.793 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_6.status 2022-04-10T01:38:26.794 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:38:26.794 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:26.794 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:26.794 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:26.795 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2022-04-10T01:38:26.795 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:38:26.841 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:38:26.841 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:38:26.843 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:38:26.843 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:38:26.843 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:38:26.843 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:38:26.844 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 134 = '' 2022-04-10T01:38:26.844 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:38:26.844 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:38:27.245 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_6.status 2022-04-10T01:38:27.245 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:27.245 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:27.246 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_6 2022-04-10T01:38:27.246 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_6.status 2022-04-10T01:38:27.246 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:38:27.246 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:27.247 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:27.247 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:27.247 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2022-04-10T01:38:27.247 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:38:27.293 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:38:27.294 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:38:27.295 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:38:27.295 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:38:27.296 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:38:27.296 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:38:27.296 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 134 = '' 2022-04-10T01:38:27.296 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:38:27.296 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:38:28.097 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_6.status 2022-04-10T01:38:28.098 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:28.098 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:28.098 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_6 2022-04-10T01:38:28.098 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_6.status 2022-04-10T01:38:28.099 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:38:28.099 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:28.099 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:28.099 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:28.099 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2022-04-10T01:38:28.100 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:38:28.137 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:38:28.140 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=134 2022-04-10T01:38:28.140 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 134 = 134 2022-04-10T01:38:28.140 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:28.141 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror image_6 snap 2022-04-10T01:38:28.141 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:28.141 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:28.141 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_6 2022-04-10T01:38:28.142 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:38:28.142 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:38:28.142 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:38:28.142 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:28.143 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:28.143 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:28.143 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:38:28.143 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:38:28.851 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2022-04-10T01:38:28.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2022-04-10T01:38:28.852 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2022-04-10T01:38:28.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi043.stderr:2022-04-10T01:38:28.850+0000 7f705d412700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 53318) UID: 1000 2022-04-10T01:38:29.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi043.stderr:daemon-helper: command failed with exit status 1 2022-04-10T01:38:29.140 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/image_6@snap 2022-04-10T01:38:29.181 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_6': 2022-04-10T01:38:29.182 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:38:29.182 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:38:29.182 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 2 2022-04-10T01:38:29.182 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 11333968702c 2022-04-10T01:38:29.183 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.11333968702c 2022-04-10T01:38:29.183 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:38:29.183 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:38:29.183 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:38:29.183 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:38:29.184 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:29:02 2022 2022-04-10T01:38:29.184 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:29:02 2022 2022-04-10T01:38:29.184 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:29:02 2022 2022-04-10T01:38:29.184 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:38:29.185 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:38:29.185 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:38:29.185 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: a2f508f1-f607-46da-a720-0b026055c5d9 2022-04-10T01:38:29.185 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:38:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror image_6 snap 2022-04-10T01:38:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:29.188 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_6 2022-04-10T01:38:29.188 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:38:29.188 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:38:29.188 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_6.export 2022-04-10T01:38:29.188 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_6.export 2022-04-10T01:38:29.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_6.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_6.export 2022-04-10T01:38:29.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export image_6@snap /tmp/tmp.rgwz74WspX/cluster2-mirror-image_6.export 2022-04-10T01:38:29.370 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:38:29.373 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export image_6@snap /tmp/tmp.rgwz74WspX/cluster1-mirror-image_6.export 2022-04-10T01:38:29.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:38:29.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-image_6.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_6.export 2022-04-10T01:38:29.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_6.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_6.export 2022-04-10T01:38:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:38:29.704 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_7 2022-04-10T01:38:29.704 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror image_7 snap 2022-04-10T01:38:29.704 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:38:29.704 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:29.704 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_7 2022-04-10T01:38:29.705 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:38:29.705 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create image_7@snap --debug-rbd=20 --debug-journaler=20 2022-04-10T01:38:31.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror image_7 2022-04-10T01:38:31.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:31.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:31.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_7 2022-04-10T01:38:31.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:38:31.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:31.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:31.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror image_7 started 2022-04-10T01:38:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_7 2022-04-10T01:38:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:38:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:38:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:38:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:31.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:31.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:38:31.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:38:32.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror image_7 started 2022-04-10T01:38:32.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:32.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:32.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_7 2022-04-10T01:38:32.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:38:32.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:38:32.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:38:32.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:38:32.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:32.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:32.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:32.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_7 2022-04-10T01:38:32.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:38:32.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:38:32.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:38:32.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:38:32.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:38:32.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:38:32.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:38:32.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:38:32.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:32.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:38:32.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:38:32.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:38:32.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:38:32.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:38:32.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:38:32.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_7 2022-04-10T01:38:32.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:38:32.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:38:32.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:38:32.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:38:32.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:38:32.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:38:32.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:38:32.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:38:32.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:38:32.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:38:32.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:38:32.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:32.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:38:32.562 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:38:32.562 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:38:32.562 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:38:32.562 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:32.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:38:32.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:38:32.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:38:32.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:32.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:38:32.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:38:32.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:38:32.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:38:32.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:38:32.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:32.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:38:32.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:38:32.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:38:32.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:38:32.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:38:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:38:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:38:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:38:32.567 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:38:32.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:38:32.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:38:32.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:38:32.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_7 2022-04-10T01:38:32.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2022-04-10T01:38:32.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:38:32.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_7 2022-04-10T01:38:32.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:1 2022-04-10T01:38:32.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:38:32.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:38:32.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:1 cluster instance 2022-04-10T01:38:32.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:1 2022-04-10T01:38:32.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:38:32.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:38:32.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:38:32.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=1 2022-04-10T01:38:32.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 1 = cluster1:1 ']' 2022-04-10T01:38:32.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:38:32.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:32.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=1 2022-04-10T01:38:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:38:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:38:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:38:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:38:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:1 cluster1 2022-04-10T01:38:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:1 2022-04-10T01:38:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:38:32.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2022-04-10T01:38:32.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:1 2022-04-10T01:38:32.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:38:32.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:38:32.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:32.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:38:32.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 1 = cluster1:1 ']' 2022-04-10T01:38:32.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:38:32.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:38:32.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=1 2022-04-10T01:38:32.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=1 2022-04-10T01:38:32.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:38:32.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:32.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:32.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:38:32.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2022-04-10T01:38:32.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:38:32.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:38:32.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:38:32.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_7 2022-04-10T01:38:32.789 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2022-04-10T01:38:32.789 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:38:32.789 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:38:32.793 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:38:32.793 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_7 2022-04-10T01:38:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:2 2022-04-10T01:38:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:38:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:38:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:2 cluster instance 2022-04-10T01:38:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:2 2022-04-10T01:38:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:38:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:38:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:38:32.796 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=2 2022-04-10T01:38:32.796 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 2 = cluster1:2 ']' 2022-04-10T01:38:32.796 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:38:32.796 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=2 2022-04-10T01:38:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:38:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:38:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:38:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:32.798 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:38:32.798 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:2 cluster1 2022-04-10T01:38:32.798 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:2 2022-04-10T01:38:32.798 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:32.799 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:38:32.799 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2022-04-10T01:38:32.799 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:2 2022-04-10T01:38:32.799 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:38:32.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:38:32.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:32.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:38:32.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 2 = cluster1:2 ']' 2022-04-10T01:38:32.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:38:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:38:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=2 2022-04-10T01:38:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=2 2022-04-10T01:38:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:38:32.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:32.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:32.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:38:32.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2022-04-10T01:38:32.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:38:32.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:38:32.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:38:32.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_7 2022-04-10T01:38:32.922 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:38:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:38:32.924 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:38:32.924 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:38:32.924 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:38:32.927 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:38:32.928 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:38:32.928 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:32.928 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_7 2022-04-10T01:38:32.928 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:38:32.929 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:38:32.929 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:32.929 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_7 2022-04-10T01:38:32.929 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:38:32.930 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:38:32.930 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:32.930 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:32.930 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:38:32.931 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:38:32.931 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_7 2022-04-10T01:38:32.931 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:38:32.931 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:38:32.931 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:32.932 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_7 2022-04-10T01:38:32.932 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:38:32.932 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:32.932 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:32.933 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:32.933 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:38:32.933 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:38:32.933 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:32.933 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-image_7.status 2022-04-10T01:38:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-image_7.status 2022-04-10T01:38:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:38:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_7.status 2022-04-10T01:38:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-image_7.status 2022-04-10T01:38:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-image_7.status 2022-04-10T01:38:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:38:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_7.status 2022-04-10T01:38:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror image_7 2022-04-10T01:38:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:38:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_7 2022-04-10T01:38:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:38:32.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:38:32.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:32.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:32.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_7 2022-04-10T01:38:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 137 2022-04-10T01:38:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_7 /tmp/tmp.rgwz74WspX/cluster2-mirror-image_7.status 2022-04-10T01:38:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:38:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_7 2022-04-10T01:38:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_7.status 2022-04-10T01:38:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:38:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:38:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_7 --format xml 2022-04-10T01:38:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:38:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:38:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=137 2022-04-10T01:38:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:38:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:38:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:38:33.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_7.status 2022-04-10T01:38:33.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:33.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:33.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_7 2022-04-10T01:38:33.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_7.status 2022-04-10T01:38:33.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:38:33.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:33.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:33.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2022-04-10T01:38:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:38:33.848 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:38:33.848 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:38:33.850 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:38:33.850 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:38:33.850 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:38:33.851 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:38:33.851 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 137 = '' 2022-04-10T01:38:33.851 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:38:33.852 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:38:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_7.status 2022-04-10T01:38:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:34.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:34.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_7 2022-04-10T01:38:34.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_7.status 2022-04-10T01:38:34.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:38:34.254 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:34.254 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:34.254 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:34.254 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2022-04-10T01:38:34.255 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:38:34.298 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:38:34.298 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:38:34.300 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:38:34.300 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:38:34.300 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:38:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:38:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 137 = '' 2022-04-10T01:38:34.302 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:38:34.302 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:38:35.103 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_7.status 2022-04-10T01:38:35.104 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:35.104 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:35.105 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_7 2022-04-10T01:38:35.105 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_7.status 2022-04-10T01:38:35.105 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:38:35.106 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:35.106 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:35.106 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:35.106 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2022-04-10T01:38:35.107 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:38:35.145 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:38:35.147 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=137 2022-04-10T01:38:35.148 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 137 = 137 2022-04-10T01:38:35.148 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:35.148 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror image_7 snap 2022-04-10T01:38:35.148 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_7 2022-04-10T01:38:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:38:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:38:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:38:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:38:35.151 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:38:36.148 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/image_7@snap 2022-04-10T01:38:36.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_7': 2022-04-10T01:38:36.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:38:36.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:38:36.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 2 2022-04-10T01:38:36.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 118d27aedad5 2022-04-10T01:38:36.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.118d27aedad5 2022-04-10T01:38:36.193 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:38:36.193 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:38:36.193 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:38:36.193 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:38:36.193 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:29:16 2022 2022-04-10T01:38:36.194 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:29:16 2022 2022-04-10T01:38:36.194 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:29:16 2022 2022-04-10T01:38:36.194 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:38:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:38:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:38:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 551577c0-7aad-4cdb-945f-ca54529923ea 2022-04-10T01:38:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:38:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror image_7 snap 2022-04-10T01:38:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_7 2022-04-10T01:38:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:38:36.200 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:38:36.200 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_7.export 2022-04-10T01:38:36.200 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_7.export 2022-04-10T01:38:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_7.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_7.export 2022-04-10T01:38:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export image_7@snap /tmp/tmp.rgwz74WspX/cluster2-mirror-image_7.export 2022-04-10T01:38:36.389 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:38:36.392 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export image_7@snap /tmp/tmp.rgwz74WspX/cluster1-mirror-image_7.export 2022-04-10T01:38:36.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:38:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-image_7.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_7.export 2022-04-10T01:38:36.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_7.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_7.export 2022-04-10T01:38:36.748 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:36.748 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:38:36.748 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_8 2022-04-10T01:38:36.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror image_8 snap 2022-04-10T01:38:36.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:38:36.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:36.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_8 2022-04-10T01:38:36.750 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:38:36.750 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create image_8@snap --debug-rbd=20 --debug-journaler=20 2022-04-10T01:38:37.853 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2022-04-10T01:38:37.854 INFO:teuthology.orchestra.run:waiting for 600 2022-04-10T01:38:37.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-04-10T01:38:37.855 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2022-04-10T01:38:37.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2022-04-10T01:38:37.855 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2022-04-10T01:38:37.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2022-04-10T01:38:37.859 INFO:tasks.rbd_mirror_thrash:waiting for 87 secs before thrashing 2022-04-10T01:38:38.557 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror image_8 2022-04-10T01:38:38.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:38.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:38.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_8 2022-04-10T01:38:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:38:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:38.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror image_8 started 2022-04-10T01:38:38.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:38.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:38.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_8 2022-04-10T01:38:38.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:38:38.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:38:38.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:38:38.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:38.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:38.562 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:38.562 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:38:38.562 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:38:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror image_8 started 2022-04-10T01:38:39.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:39.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:38:39.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_8 2022-04-10T01:38:39.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:38:39.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:38:39.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:38:39.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:38:39.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:38:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:38:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_8 2022-04-10T01:38:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:38:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:38:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:38:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:38:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:38:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:38:39.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:38:39.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:38:39.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:39.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:38:39.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:38:39.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:38:39.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:38:39.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:38:39.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:38:39.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_8 2022-04-10T01:38:39.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:38:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:38:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:38:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:38:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:38:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:38:39.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:38:39.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:38:39.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:38:39.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:38:39.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:38:39.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:39.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:38:39.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:38:39.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:38:39.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:38:39.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:39.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:38:39.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:38:39.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:38:39.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:39.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:38:39.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:38:39.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:38:39.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:38:39.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:38:39.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:39.583 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:38:39.583 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:38:39.583 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:38:39.583 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:38:39.584 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:38:39.584 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:38:39.584 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:38:39.584 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:39.584 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:38:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:38:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:38:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:38:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:38:39.587 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_8 2022-04-10T01:38:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2022-04-10T01:38:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:38:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:38:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:38:39.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_8 2022-04-10T01:38:39.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:1 2022-04-10T01:38:39.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:38:39.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:38:39.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:1 cluster instance 2022-04-10T01:38:39.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:1 2022-04-10T01:38:39.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:38:39.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:38:39.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:38:39.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=1 2022-04-10T01:38:39.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 1 = cluster1:1 ']' 2022-04-10T01:38:39.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:38:39.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:39.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=1 2022-04-10T01:38:39.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:38:39.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:38:39.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:38:39.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:39.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:38:39.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:1 cluster1 2022-04-10T01:38:39.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:1 2022-04-10T01:38:39.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:39.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:38:39.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2022-04-10T01:38:39.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:1 2022-04-10T01:38:39.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:38:39.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:38:39.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:39.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:38:39.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 1 = cluster1:1 ']' 2022-04-10T01:38:39.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:38:39.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:38:39.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=1 2022-04-10T01:38:39.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=1 2022-04-10T01:38:39.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:38:39.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:39.698 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:39.698 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:38:39.698 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2022-04-10T01:38:39.699 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:38:39.699 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:38:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:38:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_8 2022-04-10T01:38:39.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2022-04-10T01:38:39.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:38:39.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:38:39.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:38:39.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_8 2022-04-10T01:38:39.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:2 2022-04-10T01:38:39.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:38:39.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:38:39.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:2 cluster instance 2022-04-10T01:38:39.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:2 2022-04-10T01:38:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:38:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:38:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:38:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=2 2022-04-10T01:38:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 2 = cluster1:2 ']' 2022-04-10T01:38:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:38:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=2 2022-04-10T01:38:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:38:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:38:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:38:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:39.812 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:38:39.812 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:2 cluster1 2022-04-10T01:38:39.812 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:2 2022-04-10T01:38:39.812 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:39.812 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:38:39.813 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2022-04-10T01:38:39.813 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:2 2022-04-10T01:38:39.813 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:38:39.813 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:38:39.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:39.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:38:39.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 2 = cluster1:2 ']' 2022-04-10T01:38:39.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:38:39.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:38:39.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=2 2022-04-10T01:38:39.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=2 2022-04-10T01:38:39.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:38:39.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:39.816 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:39.816 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:38:39.816 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2022-04-10T01:38:39.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:38:39.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:38:39.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:38:39.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_8 2022-04-10T01:38:39.927 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2022-04-10T01:38:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:38:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:38:39.933 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:38:39.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_8 2022-04-10T01:38:39.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:3 2022-04-10T01:38:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:38:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:38:39.936 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:3 cluster instance 2022-04-10T01:38:39.936 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:3 2022-04-10T01:38:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:38:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:38:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:38:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=3 2022-04-10T01:38:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 3 = cluster1:3 ']' 2022-04-10T01:38:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:38:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=3 2022-04-10T01:38:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=3 2022-04-10T01:38:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:38:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:38:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:38:39.943 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:3 cluster1 2022-04-10T01:38:39.943 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:3 2022-04-10T01:38:39.944 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:39.944 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:38:39.944 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2022-04-10T01:38:39.945 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:3 2022-04-10T01:38:39.945 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:38:39.946 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:38:39.946 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:39.947 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=3 2022-04-10T01:38:39.947 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 3 = cluster1:3 ']' 2022-04-10T01:38:39.948 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:38:39.948 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:38:39.949 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=3 2022-04-10T01:38:39.949 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=3 2022-04-10T01:38:39.949 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:38:39.950 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:39.950 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:39.951 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:38:39.951 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2022-04-10T01:38:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:38:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:38:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:38:39.955 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_8 2022-04-10T01:38:40.053 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2022-04-10T01:38:40.054 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:38:40.054 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:38:40.056 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:38:40.056 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 1 2022-04-10T01:38:41.055 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:38:41.056 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:38:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:38:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_8 2022-04-10T01:38:41.061 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:38:41.061 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:38:41.061 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:38:41.061 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:38:41.062 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:38:41.062 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:38:41.062 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:38:41.062 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:38:41.063 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:38:41.063 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:38:41.063 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:38:41.063 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:41.064 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:38:41.064 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:38:41.064 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:38:41.064 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:38:41.064 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:41.065 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:38:41.065 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:38:41.065 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:38:41.066 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:41.066 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:38:41.066 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:38:41.066 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:38:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:38:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:38:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:41.068 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:38:41.068 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:38:41.068 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:38:41.068 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:38:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:38:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:38:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:38:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:41.070 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:38:41.070 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:38:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:38:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:38:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:38:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_8 2022-04-10T01:38:41.181 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2022-04-10T01:38:41.181 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:38:41.181 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:38:41.184 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:38:41.185 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_8 2022-04-10T01:38:41.185 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:1 2022-04-10T01:38:41.185 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:38:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:38:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:1 cluster instance 2022-04-10T01:38:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:1 2022-04-10T01:38:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:38:41.187 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:38:41.187 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:38:41.187 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=1 2022-04-10T01:38:41.187 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 1 = cluster1:1 ']' 2022-04-10T01:38:41.187 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:38:41.188 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:41.188 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=1 2022-04-10T01:38:41.188 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:38:41.188 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:38:41.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:38:41.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:41.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:38:41.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:1 cluster1 2022-04-10T01:38:41.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:1 2022-04-10T01:38:41.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:41.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:38:41.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2022-04-10T01:38:41.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:1 2022-04-10T01:38:41.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:38:41.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:38:41.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:41.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:38:41.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 1 = cluster1:1 ']' 2022-04-10T01:38:41.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:38:41.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:38:41.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=1 2022-04-10T01:38:41.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=1 2022-04-10T01:38:41.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:38:41.193 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:41.193 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:41.193 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:38:41.193 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2022-04-10T01:38:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:38:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:38:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:38:41.201 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_8 2022-04-10T01:38:41.307 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2022-04-10T01:38:41.307 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:38:41.307 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:38:41.311 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:38:41.311 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_8 2022-04-10T01:38:41.311 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:2 2022-04-10T01:38:41.311 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:38:41.312 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:38:41.312 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:2 cluster instance 2022-04-10T01:38:41.312 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:2 2022-04-10T01:38:41.312 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:38:41.313 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:38:41.313 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:38:41.313 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=2 2022-04-10T01:38:41.313 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 2 = cluster1:2 ']' 2022-04-10T01:38:41.313 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:38:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=2 2022-04-10T01:38:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:38:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:38:41.315 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:38:41.315 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:41.315 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:38:41.315 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:2 cluster1 2022-04-10T01:38:41.315 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:2 2022-04-10T01:38:41.316 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:41.316 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:38:41.316 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2022-04-10T01:38:41.317 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:2 2022-04-10T01:38:41.317 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:38:41.317 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:38:41.317 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:38:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 2 = cluster1:2 ']' 2022-04-10T01:38:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:38:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:38:41.319 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=2 2022-04-10T01:38:41.319 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=2 2022-04-10T01:38:41.319 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:38:41.319 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:41.320 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:41.320 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:38:41.320 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2022-04-10T01:38:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:38:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:38:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:38:41.327 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_8 2022-04-10T01:38:41.434 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2022-04-10T01:38:41.434 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:38:41.435 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:38:41.439 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:38:41.439 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_8 2022-04-10T01:38:41.439 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:3 2022-04-10T01:38:41.440 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:38:41.440 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:38:41.440 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:3 cluster instance 2022-04-10T01:38:41.440 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:3 2022-04-10T01:38:41.441 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:38:41.441 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:38:41.441 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:38:41.441 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=3 2022-04-10T01:38:41.441 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 3 = cluster1:3 ']' 2022-04-10T01:38:41.442 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:38:41.442 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:41.442 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=3 2022-04-10T01:38:41.442 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=3 2022-04-10T01:38:41.443 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:38:41.443 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:38:41.443 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:41.443 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:38:41.443 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:3 cluster1 2022-04-10T01:38:41.444 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:3 2022-04-10T01:38:41.444 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:41.444 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:38:41.444 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2022-04-10T01:38:41.445 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:3 2022-04-10T01:38:41.445 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:38:41.445 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:38:41.445 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:41.446 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=3 2022-04-10T01:38:41.446 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 3 = cluster1:3 ']' 2022-04-10T01:38:41.446 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:38:41.446 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:38:41.447 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=3 2022-04-10T01:38:41.447 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=3 2022-04-10T01:38:41.447 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:38:41.447 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:38:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2022-04-10T01:38:41.452 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:38:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:38:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:38:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_8 2022-04-10T01:38:41.567 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2022-04-10T01:38:41.567 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:38:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:38:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:38:41.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 2 2022-04-10T01:38:42.392 INFO:tasks.ceph.cluster1.osd.0.smithi005.stderr:2022-04-10T01:38:42.390+0000 7fb18c524700 -1 reset not still connected to 0x55fe5ead49c0 2022-04-10T01:38:42.393 INFO:tasks.ceph.cluster1.osd.0.smithi005.stderr:2022-04-10T01:38:42.390+0000 7fb18c524700 -1 reset not still connected to 0x55fe5f0f2340 2022-04-10T01:38:43.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:38:43.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:38:43.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:38:43.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_8 2022-04-10T01:38:43.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:38:43.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:38:43.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:38:43.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:38:43.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:38:43.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:38:43.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:38:43.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:38:43.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:38:43.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:38:43.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:38:43.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:43.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:38:43.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:38:43.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:38:43.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:38:43.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:43.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:38:43.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:38:43.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:38:43.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:43.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:38:43.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:38:43.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:38:43.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:38:43.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:38:43.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:43.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:38:43.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:38:43.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:38:43.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:38:43.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:38:43.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:38:43.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:38:43.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:43.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:43.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:38:43.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:38:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:38:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:38:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:38:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_8 2022-04-10T01:38:43.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2022-04-10T01:38:43.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:38:43.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:38:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:38:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_8 2022-04-10T01:38:43.705 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:1 2022-04-10T01:38:43.705 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:38:43.706 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:38:43.706 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:1 cluster instance 2022-04-10T01:38:43.706 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:1 2022-04-10T01:38:43.707 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:38:43.707 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:38:43.707 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:38:43.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=1 2022-04-10T01:38:43.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 1 = cluster1:1 ']' 2022-04-10T01:38:43.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:38:43.709 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:43.709 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=1 2022-04-10T01:38:43.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:38:43.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:38:43.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:38:43.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:43.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:38:43.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:1 cluster1 2022-04-10T01:38:43.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:1 2022-04-10T01:38:43.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:43.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:38:43.712 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2022-04-10T01:38:43.712 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:1 2022-04-10T01:38:43.712 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:38:43.712 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:38:43.712 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:43.713 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:38:43.713 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 1 = cluster1:1 ']' 2022-04-10T01:38:43.713 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:38:43.713 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:38:43.714 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=1 2022-04-10T01:38:43.714 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=1 2022-04-10T01:38:43.714 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:38:43.714 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:43.715 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:43.715 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:38:43.715 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2022-04-10T01:38:43.716 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:38:43.716 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:38:43.717 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:38:43.717 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_8 2022-04-10T01:38:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2022-04-10T01:38:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:38:43.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:38:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:38:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_8 2022-04-10T01:38:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:2 2022-04-10T01:38:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:38:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:38:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:2 cluster instance 2022-04-10T01:38:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:2 2022-04-10T01:38:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:38:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:38:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:38:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=2 2022-04-10T01:38:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 2 = cluster1:2 ']' 2022-04-10T01:38:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:38:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:43.825 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=2 2022-04-10T01:38:43.825 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:38:43.825 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:38:43.825 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:38:43.825 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:43.826 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:38:43.826 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:2 cluster1 2022-04-10T01:38:43.826 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:2 2022-04-10T01:38:43.826 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:43.827 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:38:43.827 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2022-04-10T01:38:43.827 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:2 2022-04-10T01:38:43.827 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:38:43.827 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:38:43.828 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:43.828 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:38:43.828 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 2 = cluster1:2 ']' 2022-04-10T01:38:43.828 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:38:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:38:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=2 2022-04-10T01:38:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=2 2022-04-10T01:38:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:38:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:38:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2022-04-10T01:38:43.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:38:43.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:38:43.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:38:43.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_8 2022-04-10T01:38:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2022-04-10T01:38:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:38:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:38:43.944 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:38:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_8 2022-04-10T01:38:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:3 2022-04-10T01:38:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:38:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:38:43.946 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:3 cluster instance 2022-04-10T01:38:43.946 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:3 2022-04-10T01:38:43.946 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:38:43.946 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:38:43.947 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:38:43.947 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=3 2022-04-10T01:38:43.947 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 3 = cluster1:3 ']' 2022-04-10T01:38:43.947 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:38:43.947 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=3 2022-04-10T01:38:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=3 2022-04-10T01:38:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:38:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:38:43.949 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:43.949 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:38:43.949 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:3 cluster1 2022-04-10T01:38:43.949 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:3 2022-04-10T01:38:43.949 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:43.950 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:38:43.950 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2022-04-10T01:38:43.950 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:3 2022-04-10T01:38:43.950 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:38:43.951 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:38:43.951 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:43.951 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=3 2022-04-10T01:38:43.951 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 3 = cluster1:3 ']' 2022-04-10T01:38:43.951 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:38:43.952 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:38:43.952 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=3 2022-04-10T01:38:43.952 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=3 2022-04-10T01:38:43.952 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:38:43.953 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:43.953 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:43.953 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:38:43.953 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2022-04-10T01:38:43.959 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:38:43.959 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:38:43.959 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:38:43.959 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_8 2022-04-10T01:38:44.063 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2022-04-10T01:38:44.063 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:38:44.064 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:38:44.064 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:38:44.065 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 4 2022-04-10T01:38:48.066 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:38:48.067 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:38:48.074 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:38:48.074 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_8 2022-04-10T01:38:48.075 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:38:48.075 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:38:48.075 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:38:48.076 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:38:48.076 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:38:48.077 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:38:48.077 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:38:48.078 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:38:48.078 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:38:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:38:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:38:48.080 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:48.080 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:38:48.080 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:38:48.081 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:38:48.081 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:38:48.081 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:48.082 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:38:48.082 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:38:48.082 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:38:48.083 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:48.083 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:38:48.083 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:38:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:38:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:38:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:38:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:38:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:38:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:38:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:38:48.086 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:38:48.086 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:38:48.086 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:38:48.086 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:48.087 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:48.087 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:38:48.087 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:38:48.088 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:38:48.088 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:38:48.089 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:38:48.089 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_8 2022-04-10T01:38:48.196 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2022-04-10T01:38:48.196 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:38:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:38:48.200 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:38:48.200 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_8 2022-04-10T01:38:48.201 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:1 2022-04-10T01:38:48.201 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:38:48.201 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:38:48.201 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:1 cluster instance 2022-04-10T01:38:48.202 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:1 2022-04-10T01:38:48.202 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:38:48.202 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:38:48.202 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:38:48.202 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=1 2022-04-10T01:38:48.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 1 = cluster1:1 ']' 2022-04-10T01:38:48.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:38:48.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:48.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=1 2022-04-10T01:38:48.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:38:48.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:38:48.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:38:48.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:48.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:38:48.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:1 cluster1 2022-04-10T01:38:48.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:1 2022-04-10T01:38:48.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:48.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:38:48.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2022-04-10T01:38:48.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:1 2022-04-10T01:38:48.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:38:48.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:38:48.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:48.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:38:48.207 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 1 = cluster1:1 ']' 2022-04-10T01:38:48.207 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:38:48.207 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:38:48.207 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=1 2022-04-10T01:38:48.207 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=1 2022-04-10T01:38:48.208 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:38:48.208 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:48.208 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:48.208 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:38:48.208 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2022-04-10T01:38:48.215 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:38:48.216 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:38:48.216 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:38:48.216 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_8 2022-04-10T01:38:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2022-04-10T01:38:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:38:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:38:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:38:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_8 2022-04-10T01:38:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:2 2022-04-10T01:38:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:38:48.326 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:38:48.326 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:2 cluster instance 2022-04-10T01:38:48.326 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:2 2022-04-10T01:38:48.326 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:38:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:38:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:38:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=2 2022-04-10T01:38:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 2 = cluster1:2 ']' 2022-04-10T01:38:48.328 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:38:48.328 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:48.328 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=2 2022-04-10T01:38:48.328 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:38:48.328 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:38:48.329 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:38:48.329 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:48.329 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:38:48.329 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:2 cluster1 2022-04-10T01:38:48.329 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:2 2022-04-10T01:38:48.330 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:48.330 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:38:48.330 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2022-04-10T01:38:48.330 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:2 2022-04-10T01:38:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:38:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:38:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:38:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 2 = cluster1:2 ']' 2022-04-10T01:38:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:38:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:38:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=2 2022-04-10T01:38:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=2 2022-04-10T01:38:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:38:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:38:48.334 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2022-04-10T01:38:48.339 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:38:48.339 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:38:48.340 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:38:48.340 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_8 2022-04-10T01:38:48.445 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2022-04-10T01:38:48.445 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:38:48.446 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:38:48.450 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:38:48.450 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_8 2022-04-10T01:38:48.451 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:3 2022-04-10T01:38:48.451 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:38:48.451 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:38:48.451 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:3 cluster instance 2022-04-10T01:38:48.452 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:3 2022-04-10T01:38:48.452 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:38:48.452 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:38:48.452 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:38:48.453 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=3 2022-04-10T01:38:48.453 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 3 = cluster1:3 ']' 2022-04-10T01:38:48.453 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:38:48.453 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:48.453 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=3 2022-04-10T01:38:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=3 2022-04-10T01:38:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:38:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:38:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:38:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:3 cluster1 2022-04-10T01:38:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:3 2022-04-10T01:38:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:38:48.456 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2022-04-10T01:38:48.456 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:3 2022-04-10T01:38:48.456 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:38:48.456 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:38:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=3 2022-04-10T01:38:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 3 = cluster1:3 ']' 2022-04-10T01:38:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:38:48.458 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:38:48.458 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=3 2022-04-10T01:38:48.458 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=3 2022-04-10T01:38:48.458 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:38:48.459 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:48.459 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:48.459 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:38:48.459 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2022-04-10T01:38:48.464 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:38:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:38:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:38:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_8 2022-04-10T01:38:48.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2022-04-10T01:38:48.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:38:48.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:38:48.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:38:48.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 8 2022-04-10T01:38:56.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:38:56.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:38:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:38:56.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_8 2022-04-10T01:38:56.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:38:56.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:38:56.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:38:56.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:38:56.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:38:56.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:38:56.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:38:56.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:38:56.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:38:56.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:38:56.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:38:56.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:38:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:38:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:38:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:38:56.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:56.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:38:56.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:38:56.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:38:56.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:56.583 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:38:56.583 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:38:56.583 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:38:56.583 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:38:56.583 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:38:56.584 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:56.584 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:38:56.584 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:38:56.584 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:38:56.585 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:38:56.585 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:38:56.585 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:38:56.585 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:38:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:38:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:38:56.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:38:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:38:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:38:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_8 2022-04-10T01:38:56.701 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2022-04-10T01:38:56.702 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:38:56.702 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:38:56.706 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:38:56.707 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_8 2022-04-10T01:38:56.707 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:1 2022-04-10T01:38:56.707 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:38:56.707 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:38:56.707 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:1 cluster instance 2022-04-10T01:38:56.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:1 2022-04-10T01:38:56.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:38:56.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:38:56.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:38:56.709 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=1 2022-04-10T01:38:56.709 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 1 = cluster1:1 ']' 2022-04-10T01:38:56.709 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:38:56.709 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:56.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=1 2022-04-10T01:38:56.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:38:56.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:38:56.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:38:56.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:38:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:1 cluster1 2022-04-10T01:38:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:1 2022-04-10T01:38:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:56.712 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:38:56.712 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2022-04-10T01:38:56.712 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:1 2022-04-10T01:38:56.712 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:38:56.712 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:38:56.713 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:56.713 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:38:56.713 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 1 = cluster1:1 ']' 2022-04-10T01:38:56.713 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:38:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:38:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=1 2022-04-10T01:38:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=1 2022-04-10T01:38:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:38:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:56.715 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:56.715 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:38:56.715 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2022-04-10T01:38:56.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:38:56.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:38:56.722 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:38:56.722 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_8 2022-04-10T01:38:56.824 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2022-04-10T01:38:56.824 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:38:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:38:56.828 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:38:56.828 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_8 2022-04-10T01:38:56.829 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:2 2022-04-10T01:38:56.829 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:38:56.829 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:38:56.829 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:2 cluster instance 2022-04-10T01:38:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:2 2022-04-10T01:38:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:38:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:38:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:38:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=2 2022-04-10T01:38:56.831 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 2 = cluster1:2 ']' 2022-04-10T01:38:56.831 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:38:56.831 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:56.831 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=2 2022-04-10T01:38:56.832 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:38:56.832 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:38:56.832 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:38:56.832 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:56.833 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:38:56.833 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:2 cluster1 2022-04-10T01:38:56.833 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:2 2022-04-10T01:38:56.833 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:56.833 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:38:56.834 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2022-04-10T01:38:56.834 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:2 2022-04-10T01:38:56.834 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:38:56.834 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:38:56.834 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:56.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:38:56.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 2 = cluster1:2 ']' 2022-04-10T01:38:56.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:38:56.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:38:56.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=2 2022-04-10T01:38:56.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=2 2022-04-10T01:38:56.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:38:56.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:56.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:56.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:38:56.837 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2022-04-10T01:38:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:38:56.844 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:38:56.844 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:38:56.844 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_8 2022-04-10T01:38:56.947 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2022-04-10T01:38:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:38:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:38:56.952 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:38:56.952 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_8 2022-04-10T01:38:56.952 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:3 2022-04-10T01:38:56.953 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:38:56.953 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:38:56.953 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:3 cluster instance 2022-04-10T01:38:56.953 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:3 2022-04-10T01:38:56.953 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:38:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:38:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:38:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=3 2022-04-10T01:38:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 3 = cluster1:3 ']' 2022-04-10T01:38:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:38:56.955 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:56.955 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=3 2022-04-10T01:38:56.955 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=3 2022-04-10T01:38:56.955 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:38:56.956 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:38:56.956 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:56.956 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:38:56.956 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:3 cluster1 2022-04-10T01:38:56.956 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:3 2022-04-10T01:38:56.957 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:56.957 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:38:56.957 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2022-04-10T01:38:56.957 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:3 2022-04-10T01:38:56.958 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:38:56.958 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:38:56.958 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:38:56.958 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=3 2022-04-10T01:38:56.958 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 3 = cluster1:3 ']' 2022-04-10T01:38:56.959 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:38:56.959 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:38:56.959 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=3 2022-04-10T01:38:56.959 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=3 2022-04-10T01:38:56.959 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:38:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:38:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:38:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:38:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2022-04-10T01:38:56.967 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:38:56.967 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:38:56.967 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:38:56.967 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_8 2022-04-10T01:38:57.070 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2022-04-10T01:38:57.071 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:38:57.071 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:38:57.072 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:38:57.072 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 8 2022-04-10T01:39:05.073 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:39:05.073 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:39:05.077 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:39:05.077 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_8 2022-04-10T01:39:05.078 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:39:05.078 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:39:05.078 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:39:05.078 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:39:05.079 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:39:05.079 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:39:05.079 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:39:05.079 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:39:05.079 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:39:05.080 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:39:05.080 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:39:05.080 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:39:05.081 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:39:05.081 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:39:05.081 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:39:05.081 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:39:05.081 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:05.082 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:39:05.082 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:39:05.082 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:39:05.082 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:39:05.083 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:39:05.083 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:39:05.083 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:39:05.083 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:39:05.083 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:39:05.084 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:39:05.084 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:39:05.084 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:39:05.084 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:39:05.085 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:39:05.085 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:39:05.085 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:39:05.085 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:39:05.085 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:39:05.086 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:05.086 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:39:05.086 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:39:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:39:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:39:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:39:05.097 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_8 2022-04-10T01:39:05.210 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2022-04-10T01:39:05.210 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:39:05.210 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:39:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:39:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_8 2022-04-10T01:39:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:1 2022-04-10T01:39:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:39:05.216 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:39:05.216 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:1 cluster instance 2022-04-10T01:39:05.216 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:1 2022-04-10T01:39:05.216 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:39:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:39:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:39:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=1 2022-04-10T01:39:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 1 = cluster1:1 ']' 2022-04-10T01:39:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:39:05.218 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:39:05.218 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=1 2022-04-10T01:39:05.218 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:39:05.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:39:05.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:39:05.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:05.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:39:05.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:1 cluster1 2022-04-10T01:39:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:1 2022-04-10T01:39:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:39:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:39:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2022-04-10T01:39:05.221 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:1 2022-04-10T01:39:05.222 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:39:05.222 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:39:05.222 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:39:05.222 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:39:05.223 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 1 = cluster1:1 ']' 2022-04-10T01:39:05.223 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:39:05.223 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:39:05.223 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=1 2022-04-10T01:39:05.223 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=1 2022-04-10T01:39:05.224 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:39:05.224 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:39:05.224 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:05.225 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:39:05.225 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2022-04-10T01:39:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:39:05.229 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:39:05.229 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:39:05.229 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_8 2022-04-10T01:39:05.335 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2022-04-10T01:39:05.336 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:39:05.336 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:39:05.340 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:39:05.340 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_8 2022-04-10T01:39:05.340 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:2 2022-04-10T01:39:05.340 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:39:05.341 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:39:05.341 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:2 cluster instance 2022-04-10T01:39:05.341 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:2 2022-04-10T01:39:05.341 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:39:05.342 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:39:05.342 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:39:05.342 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=2 2022-04-10T01:39:05.342 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 2 = cluster1:2 ']' 2022-04-10T01:39:05.343 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:39:05.343 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:39:05.343 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=2 2022-04-10T01:39:05.343 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:39:05.343 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:39:05.344 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:39:05.344 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:05.344 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:39:05.344 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:2 cluster1 2022-04-10T01:39:05.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:2 2022-04-10T01:39:05.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:39:05.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:39:05.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2022-04-10T01:39:05.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:2 2022-04-10T01:39:05.346 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:39:05.346 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:39:05.346 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:39:05.346 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:39:05.347 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 2 = cluster1:2 ']' 2022-04-10T01:39:05.347 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:39:05.347 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:39:05.347 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=2 2022-04-10T01:39:05.348 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=2 2022-04-10T01:39:05.348 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:39:05.348 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:39:05.348 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:05.348 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:39:05.349 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2022-04-10T01:39:05.354 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:39:05.355 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:39:05.355 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:39:05.355 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_8 2022-04-10T01:39:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:39:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:39:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:39:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:39:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:39:05.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:39:05.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:39:05.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:05.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_8 2022-04-10T01:39:05.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:39:05.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:05.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:05.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_8 2022-04-10T01:39:05.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:39:05.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:05.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:05.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:05.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:39:05.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:39:05.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_8 2022-04-10T01:39:05.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:39:05.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:39:05.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:05.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_8 2022-04-10T01:39:05.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:05.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:05.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:05.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:05.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:39:05.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:05.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:05.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:05.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-image_8.status 2022-04-10T01:39:05.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-image_8.status 2022-04-10T01:39:05.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:39:05.483 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_8.status 2022-04-10T01:39:05.483 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-image_8.status 2022-04-10T01:39:05.483 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-image_8.status 2022-04-10T01:39:05.483 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:39:05.484 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_8.status 2022-04-10T01:39:05.484 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror image_8 2022-04-10T01:39:05.484 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:05.484 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:05.484 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_8 2022-04-10T01:39:05.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:39:05.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:05.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:05.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:05.486 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_8 2022-04-10T01:39:05.583 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 140 2022-04-10T01:39:05.592 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_8 /tmp/tmp.rgwz74WspX/cluster2-mirror-image_8.status 2022-04-10T01:39:05.592 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:05.592 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:05.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_8 2022-04-10T01:39:05.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_8.status 2022-04-10T01:39:05.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:39:05.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:05.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:05.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:05.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_8 --format xml 2022-04-10T01:39:05.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:39:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:39:05.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=140 2022-04-10T01:39:05.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:39:05.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:39:05.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:39:05.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_8.status 2022-04-10T01:39:05.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:05.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:05.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_8 2022-04-10T01:39:05.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_8.status 2022-04-10T01:39:05.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:05.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:05.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:05.837 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:05.837 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2022-04-10T01:39:05.837 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:39:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:39:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:39:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:39:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:39:05.879 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:39:05.879 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:39:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 140 = '' 2022-04-10T01:39:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:39:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:39:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_8.status 2022-04-10T01:39:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_8 2022-04-10T01:39:06.282 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_8.status 2022-04-10T01:39:06.282 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:06.282 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:06.282 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:06.283 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:06.283 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2022-04-10T01:39:06.283 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:39:06.325 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:39:06.325 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:39:06.326 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:39:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:39:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:39:06.328 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:39:06.328 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 140 = '' 2022-04-10T01:39:06.328 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:39:06.328 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:39:07.129 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_8.status 2022-04-10T01:39:07.129 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:07.130 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:07.130 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_8 2022-04-10T01:39:07.130 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_8.status 2022-04-10T01:39:07.130 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:07.132 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2022-04-10T01:39:07.132 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:39:07.169 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:39:07.172 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=140 2022-04-10T01:39:07.172 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 140 = 140 2022-04-10T01:39:07.172 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror image_8 snap 2022-04-10T01:39:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_8 2022-04-10T01:39:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:39:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:39:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:39:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:39:08.172 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/image_8@snap 2022-04-10T01:39:08.214 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_8': 2022-04-10T01:39:08.215 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:39:08.215 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:39:08.215 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 2 2022-04-10T01:39:08.216 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 12873e30f0ca 2022-04-10T01:39:08.216 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.12873e30f0ca 2022-04-10T01:39:08.216 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:39:08.216 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:39:08.216 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:39:08.217 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:39:08.217 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:29:23 2022 2022-04-10T01:39:08.217 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:29:23 2022 2022-04-10T01:39:08.217 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:29:23 2022 2022-04-10T01:39:08.218 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:39:08.218 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:39:08.218 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:39:08.218 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 77c1be68-fcf7-4bbb-bf06-177e6f3d4112 2022-04-10T01:39:08.218 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:39:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror image_8 snap 2022-04-10T01:39:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:08.221 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_8 2022-04-10T01:39:08.221 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:39:08.221 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:39:08.221 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_8.export 2022-04-10T01:39:08.222 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_8.export 2022-04-10T01:39:08.222 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_8.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_8.export 2022-04-10T01:39:08.222 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export image_8@snap /tmp/tmp.rgwz74WspX/cluster2-mirror-image_8.export 2022-04-10T01:39:08.403 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:39:08.406 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export image_8@snap /tmp/tmp.rgwz74WspX/cluster1-mirror-image_8.export 2022-04-10T01:39:08.618 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:39:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-image_8.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_8.export 2022-04-10T01:39:08.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_8.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_8.export 2022-04-10T01:39:08.724 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:08.724 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:39:08.724 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_9 2022-04-10T01:39:08.724 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror image_9 snap 2022-04-10T01:39:08.725 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:08.725 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:08.725 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_9 2022-04-10T01:39:08.725 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:39:08.726 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create image_9@snap --debug-rbd=20 --debug-journaler=20 2022-04-10T01:39:10.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror image_9 2022-04-10T01:39:10.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:10.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:10.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_9 2022-04-10T01:39:10.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:10.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:10.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:10.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:10.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror image_9 started 2022-04-10T01:39:10.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:10.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:10.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_9 2022-04-10T01:39:10.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:39:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:39:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:39:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:39:11.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror image_9 started 2022-04-10T01:39:11.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:11.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:11.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_9 2022-04-10T01:39:11.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:39:11.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:39:11.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:39:11.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:11.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:11.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:11.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:11.600 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_9 2022-04-10T01:39:11.600 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:39:11.600 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:39:11.600 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:39:11.601 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:39:11.601 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:39:11.601 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:39:11.601 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:39:11.601 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:39:11.602 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:11.602 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:39:11.602 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:39:11.602 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:39:11.603 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:39:11.603 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:39:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:39:11.608 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_9 2022-04-10T01:39:11.608 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:39:11.608 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:39:11.608 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:39:11.608 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:39:11.609 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:39:11.609 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:39:11.609 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:39:11.609 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:39:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:39:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:39:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:39:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:39:11.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:39:11.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:39:11.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:39:11.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:39:11.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:11.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:39:11.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:39:11.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:39:11.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:39:11.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:39:11.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:39:11.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:39:11.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:39:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:39:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:39:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:39:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:39:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:39:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:39:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:39:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:39:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:39:11.616 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:39:11.616 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:11.616 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:39:11.616 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:39:11.619 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:39:11.619 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:39:11.619 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:39:11.619 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_9 2022-04-10T01:39:11.732 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2022-04-10T01:39:11.732 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:39:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:39:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:39:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_9 2022-04-10T01:39:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:1 2022-04-10T01:39:11.738 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:39:11.738 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:39:11.738 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:1 cluster instance 2022-04-10T01:39:11.738 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:1 2022-04-10T01:39:11.739 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:39:11.739 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:39:11.739 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:39:11.739 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=1 2022-04-10T01:39:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 1 = cluster1:1 ']' 2022-04-10T01:39:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:39:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:39:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=1 2022-04-10T01:39:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:39:11.741 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:39:11.741 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:39:11.741 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:11.741 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:39:11.742 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:1 cluster1 2022-04-10T01:39:11.742 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:1 2022-04-10T01:39:11.742 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:39:11.742 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:39:11.742 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2022-04-10T01:39:11.743 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:1 2022-04-10T01:39:11.743 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:39:11.743 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:39:11.743 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:39:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:39:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 1 = cluster1:1 ']' 2022-04-10T01:39:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:39:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:39:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=1 2022-04-10T01:39:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=1 2022-04-10T01:39:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:39:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:39:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:39:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2022-04-10T01:39:11.752 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:39:11.752 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:39:11.752 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:39:11.753 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_9 2022-04-10T01:39:11.866 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:39:11.867 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:39:11.867 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:39:11.868 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:39:11.868 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:39:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:39:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:39:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_9 2022-04-10T01:39:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:39:11.874 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:11.874 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:11.874 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_9 2022-04-10T01:39:11.874 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:39:11.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:11.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:11.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:11.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:39:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:39:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_9 2022-04-10T01:39:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:39:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:39:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:11.877 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_9 2022-04-10T01:39:11.877 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:11.877 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:11.877 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:39:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:11.879 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:11.879 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-image_9.status 2022-04-10T01:39:11.879 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-image_9.status 2022-04-10T01:39:11.879 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:39:11.879 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_9.status 2022-04-10T01:39:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-image_9.status 2022-04-10T01:39:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-image_9.status 2022-04-10T01:39:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:39:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_9.status 2022-04-10T01:39:11.881 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror image_9 2022-04-10T01:39:11.881 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:11.881 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:11.882 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_9 2022-04-10T01:39:11.882 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:39:11.882 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:11.883 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:11.883 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:11.883 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_9 2022-04-10T01:39:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 143 2022-04-10T01:39:12.610 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_9 /tmp/tmp.rgwz74WspX/cluster2-mirror-image_9.status 2022-04-10T01:39:12.610 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:12.610 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_9 2022-04-10T01:39:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_9.status 2022-04-10T01:39:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:39:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:12.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:12.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:12.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_9 --format xml 2022-04-10T01:39:12.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:39:12.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:39:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=143 2022-04-10T01:39:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:39:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:39:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:39:12.855 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_9.status 2022-04-10T01:39:12.856 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:12.856 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:12.856 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_9 2022-04-10T01:39:12.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_9.status 2022-04-10T01:39:12.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:12.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:12.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:12.858 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:12.858 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2022-04-10T01:39:12.858 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:39:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:39:12.901 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:39:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:39:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:39:12.903 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:39:12.903 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:39:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 143 = '' 2022-04-10T01:39:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:39:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:39:13.305 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_9.status 2022-04-10T01:39:13.305 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:13.305 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_9 2022-04-10T01:39:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_9.status 2022-04-10T01:39:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2022-04-10T01:39:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:39:13.352 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:39:13.353 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:39:13.354 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:39:13.354 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:39:13.355 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:39:13.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:39:13.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 143 = '' 2022-04-10T01:39:13.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:39:13.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:39:14.157 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_9.status 2022-04-10T01:39:14.157 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:14.158 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:14.158 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_9 2022-04-10T01:39:14.158 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_9.status 2022-04-10T01:39:14.158 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:14.159 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:14.159 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:14.159 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:14.159 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2022-04-10T01:39:14.159 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:39:14.199 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:39:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=143 2022-04-10T01:39:14.202 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 143 = 143 2022-04-10T01:39:14.202 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:14.202 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror image_9 snap 2022-04-10T01:39:14.202 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:14.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:14.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_9 2022-04-10T01:39:14.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:39:14.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:39:14.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:14.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:14.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:14.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:14.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:39:14.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:39:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/image_9@snap 2022-04-10T01:39:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_9': 2022-04-10T01:39:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:39:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:39:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 2 2022-04-10T01:39:15.248 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 12a743ea0893 2022-04-10T01:39:15.248 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.12a743ea0893 2022-04-10T01:39:15.248 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:39:15.248 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:39:15.249 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:39:15.249 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:39:15.249 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:29:37 2022 2022-04-10T01:39:15.249 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:29:37 2022 2022-04-10T01:39:15.249 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:29:37 2022 2022-04-10T01:39:15.250 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:39:15.250 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:39:15.250 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:39:15.250 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 0661b513-be8b-4e37-9a6b-b32c1a1daf4e 2022-04-10T01:39:15.251 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:39:15.252 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:15.252 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror image_9 snap 2022-04-10T01:39:15.252 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_9 2022-04-10T01:39:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:39:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:39:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_9.export 2022-04-10T01:39:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_9.export 2022-04-10T01:39:15.254 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_9.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_9.export 2022-04-10T01:39:15.254 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export image_9@snap /tmp/tmp.rgwz74WspX/cluster2-mirror-image_9.export 2022-04-10T01:39:15.440 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:39:15.443 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export image_9@snap /tmp/tmp.rgwz74WspX/cluster1-mirror-image_9.export 2022-04-10T01:39:15.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:39:15.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-image_9.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_9.export 2022-04-10T01:39:15.732 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_9.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_9.export 2022-04-10T01:39:15.770 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:15.770 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:39:15.770 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_10 2022-04-10T01:39:15.771 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror image_10 snap 2022-04-10T01:39:15.771 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:15.771 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:15.771 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_10 2022-04-10T01:39:15.771 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:39:15.772 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create image_10@snap --debug-rbd=20 --debug-journaler=20 2022-04-10T01:39:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror image_10 2022-04-10T01:39:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:17.618 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_10 2022-04-10T01:39:17.618 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:17.618 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:17.618 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:17.619 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:17.619 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror image_10 started 2022-04-10T01:39:17.619 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:17.619 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_10 2022-04-10T01:39:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:39:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:39:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:17.621 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:17.621 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:17.621 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:39:17.621 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:39:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror image_10 started 2022-04-10T01:39:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_10 2022-04-10T01:39:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:39:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:39:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:39:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_10 2022-04-10T01:39:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:39:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:39:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:39:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:39:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:39:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:39:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:39:18.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:39:18.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:18.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:39:18.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:39:18.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:39:18.625 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:39:18.625 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:39:18.629 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:39:18.629 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_10 2022-04-10T01:39:18.629 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:39:18.629 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:39:18.630 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:39:18.630 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:39:18.630 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:39:18.630 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:39:18.630 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:39:18.631 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:39:18.631 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:39:18.631 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:39:18.631 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:39:18.632 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:39:18.632 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:39:18.632 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:39:18.632 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:39:18.632 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:39:18.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:18.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:39:18.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:39:18.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:39:18.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:39:18.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:39:18.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:39:18.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:39:18.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:39:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:39:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:39:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:39:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:39:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:39:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:39:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:39:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:39:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:39:18.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:39:18.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:18.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:39:18.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:39:18.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:39:18.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:39:18.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:39:18.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_10 2022-04-10T01:39:18.754 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2022-04-10T01:39:18.755 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:39:18.755 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:39:18.758 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:39:18.759 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_10 2022-04-10T01:39:18.759 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:1 2022-04-10T01:39:18.759 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:39:18.759 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:39:18.760 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:1 cluster instance 2022-04-10T01:39:18.760 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:1 2022-04-10T01:39:18.760 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:39:18.760 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:39:18.760 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:39:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=1 2022-04-10T01:39:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 1 = cluster1:1 ']' 2022-04-10T01:39:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:39:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:39:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=1 2022-04-10T01:39:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:39:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:39:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:39:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:39:18.763 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:1 cluster1 2022-04-10T01:39:18.763 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:1 2022-04-10T01:39:18.763 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:39:18.763 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:39:18.763 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2022-04-10T01:39:18.764 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:1 2022-04-10T01:39:18.764 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:39:18.764 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:39:18.764 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:39:18.765 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:39:18.765 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 1 = cluster1:1 ']' 2022-04-10T01:39:18.765 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:39:18.765 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:39:18.765 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=1 2022-04-10T01:39:18.766 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=1 2022-04-10T01:39:18.766 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:39:18.766 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:39:18.766 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:18.766 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:39:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2022-04-10T01:39:18.773 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:39:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:39:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:39:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_10 2022-04-10T01:39:18.879 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2022-04-10T01:39:18.879 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:39:18.879 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:39:18.883 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:39:18.883 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_10 2022-04-10T01:39:18.883 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:2 2022-04-10T01:39:18.884 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:39:18.884 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:39:18.884 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:2 cluster instance 2022-04-10T01:39:18.884 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:2 2022-04-10T01:39:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:39:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:39:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:39:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=2 2022-04-10T01:39:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 2 = cluster1:2 ']' 2022-04-10T01:39:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:39:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:39:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=2 2022-04-10T01:39:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:39:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:39:18.887 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:39:18.887 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:18.887 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:39:18.887 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:2 cluster1 2022-04-10T01:39:18.888 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:2 2022-04-10T01:39:18.888 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:39:18.888 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:39:18.888 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2022-04-10T01:39:18.889 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:2 2022-04-10T01:39:18.889 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:39:18.889 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:39:18.889 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:39:18.889 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:39:18.890 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 2 = cluster1:2 ']' 2022-04-10T01:39:18.890 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:39:18.890 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:39:18.890 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=2 2022-04-10T01:39:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=2 2022-04-10T01:39:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:39:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:39:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:39:18.892 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2022-04-10T01:39:18.897 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:39:18.898 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:39:18.898 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:39:18.898 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_10 2022-04-10T01:39:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:39:19.014 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:39:19.014 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:39:19.014 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:39:19.015 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:39:19.019 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:39:19.019 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:39:19.019 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:19.020 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_10 2022-04-10T01:39:19.020 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:39:19.020 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:19.020 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:19.020 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_10 2022-04-10T01:39:19.021 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:39:19.021 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:19.021 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:19.021 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:19.022 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:39:19.022 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:39:19.022 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_10 2022-04-10T01:39:19.022 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:39:19.023 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:39:19.023 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:19.023 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_10 2022-04-10T01:39:19.023 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:19.023 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:19.024 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:19.024 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:19.024 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:39:19.024 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:19.025 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:19.025 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:19.025 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-image_10.status 2022-04-10T01:39:19.025 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-image_10.status 2022-04-10T01:39:19.026 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:39:19.026 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_10.status 2022-04-10T01:39:19.026 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-image_10.status 2022-04-10T01:39:19.026 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-image_10.status 2022-04-10T01:39:19.026 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:39:19.027 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_10.status 2022-04-10T01:39:19.027 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror image_10 2022-04-10T01:39:19.027 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:19.027 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:19.028 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_10 2022-04-10T01:39:19.028 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:39:19.028 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:19.028 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:19.029 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:19.029 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_10 2022-04-10T01:39:19.616 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 146 2022-04-10T01:39:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_10 /tmp/tmp.rgwz74WspX/cluster2-mirror-image_10.status 2022-04-10T01:39:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_10 2022-04-10T01:39:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_10.status 2022-04-10T01:39:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:39:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:19.630 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:19.630 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_10 --format xml 2022-04-10T01:39:19.630 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:39:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:39:19.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=146 2022-04-10T01:39:19.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:39:19.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:39:19.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:39:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_10.status 2022-04-10T01:39:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_10 2022-04-10T01:39:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_10.status 2022-04-10T01:39:19.876 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:19.876 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:19.876 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:19.876 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:19.877 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2022-04-10T01:39:19.877 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:39:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:39:19.920 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:39:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:39:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:39:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:39:19.922 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:39:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 146 = '' 2022-04-10T01:39:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:39:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:39:20.323 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_10.status 2022-04-10T01:39:20.324 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:20.324 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:20.324 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_10 2022-04-10T01:39:20.325 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_10.status 2022-04-10T01:39:20.325 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:20.325 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:20.325 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:20.325 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:20.326 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2022-04-10T01:39:20.326 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:39:20.372 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:39:20.372 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:39:20.374 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:39:20.374 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:39:20.374 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:39:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:39:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 146 = '' 2022-04-10T01:39:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:39:20.376 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:39:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_10.status 2022-04-10T01:39:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_10 2022-04-10T01:39:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_10.status 2022-04-10T01:39:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:21.179 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2022-04-10T01:39:21.179 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:39:21.218 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:39:21.221 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=146 2022-04-10T01:39:21.222 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 146 = 146 2022-04-10T01:39:21.222 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:21.222 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror image_10 snap 2022-04-10T01:39:21.222 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:21.223 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:21.223 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_10 2022-04-10T01:39:21.223 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:39:21.223 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:39:21.224 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:21.224 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:21.224 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:21.224 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:21.225 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:39:21.225 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:39:22.222 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/image_10@snap 2022-04-10T01:39:22.264 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_10': 2022-04-10T01:39:22.264 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:39:22.265 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:39:22.265 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 2 2022-04-10T01:39:22.265 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 12a720b1701c 2022-04-10T01:39:22.265 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.12a720b1701c 2022-04-10T01:39:22.266 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:39:22.266 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:39:22.266 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:39:22.266 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:39:22.267 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:29:47 2022 2022-04-10T01:39:22.267 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:29:47 2022 2022-04-10T01:39:22.267 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:29:47 2022 2022-04-10T01:39:22.267 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:39:22.268 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:39:22.268 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:39:22.268 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: e9d2f488-240a-4db7-8e1f-b458c15d12a8 2022-04-10T01:39:22.268 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:39:22.270 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:22.270 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror image_10 snap 2022-04-10T01:39:22.270 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:22.271 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_10 2022-04-10T01:39:22.271 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:39:22.271 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:39:22.271 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_10.export 2022-04-10T01:39:22.271 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_10.export 2022-04-10T01:39:22.272 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_10.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_10.export 2022-04-10T01:39:22.272 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export image_10@snap /tmp/tmp.rgwz74WspX/cluster2-mirror-image_10.export 2022-04-10T01:39:22.453 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:39:22.456 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export image_10@snap /tmp/tmp.rgwz74WspX/cluster1-mirror-image_10.export 2022-04-10T01:39:22.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:39:22.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-image_10.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_10.export 2022-04-10T01:39:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_10.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_10.export 2022-04-10T01:39:22.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:22.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:39:22.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_11 2022-04-10T01:39:22.769 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror image_11 snap 2022-04-10T01:39:22.769 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:22.769 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:22.769 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_11 2022-04-10T01:39:22.770 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:39:22.770 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create image_11@snap --debug-rbd=20 --debug-journaler=20 2022-04-10T01:39:24.493 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror image_11 2022-04-10T01:39:24.493 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:24.493 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:24.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_11 2022-04-10T01:39:24.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:24.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:24.495 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:24.495 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:24.495 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror image_11 started 2022-04-10T01:39:24.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:24.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:24.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_11 2022-04-10T01:39:24.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:39:24.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:39:24.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:24.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:39:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:39:25.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror image_11 started 2022-04-10T01:39:25.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:25.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:25.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_11 2022-04-10T01:39:25.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:39:25.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:39:25.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:39:25.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:25.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:25.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:25.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:25.500 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_11 2022-04-10T01:39:25.500 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:39:25.500 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:39:25.500 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:39:25.500 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:39:25.501 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:39:25.501 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:39:25.501 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:39:25.501 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:39:25.502 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:25.502 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:39:25.502 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:39:25.502 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:39:25.502 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:39:25.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:39:25.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:39:25.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_11 2022-04-10T01:39:25.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:39:25.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:39:25.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:39:25.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:39:25.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:39:25.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:39:25.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:39:25.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:39:25.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:39:25.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:39:25.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:39:25.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:39:25.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:39:25.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:39:25.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:39:25.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:39:25.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:25.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:39:25.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:39:25.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:39:25.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:39:25.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:39:25.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:39:25.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:39:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:39:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:39:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:39:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:39:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:39:25.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:39:25.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:39:25.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:39:25.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:39:25.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:39:25.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:39:25.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:25.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:39:25.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:39:25.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:39:25.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:39:25.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:39:25.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_11 2022-04-10T01:39:25.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2022-04-10T01:39:25.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:39:25.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:39:25.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:39:25.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_11 2022-04-10T01:39:25.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:1 2022-04-10T01:39:25.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:39:25.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:39:25.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:1 cluster instance 2022-04-10T01:39:25.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:1 2022-04-10T01:39:25.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:39:25.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:39:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:39:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=1 2022-04-10T01:39:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 1 = cluster1:1 ']' 2022-04-10T01:39:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:39:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:39:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=1 2022-04-10T01:39:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:39:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:39:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:39:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:39:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:1 cluster1 2022-04-10T01:39:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:1 2022-04-10T01:39:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:39:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:39:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2022-04-10T01:39:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:1 2022-04-10T01:39:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:39:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:39:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:39:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:39:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 1 = cluster1:1 ']' 2022-04-10T01:39:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:39:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:39:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=1 2022-04-10T01:39:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=1 2022-04-10T01:39:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:39:25.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:39:25.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:25.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:39:25.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2022-04-10T01:39:25.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:39:25.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:39:25.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:39:25.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_11 2022-04-10T01:39:25.757 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2022-04-10T01:39:25.758 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:39:25.758 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:39:25.762 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:39:25.763 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_11 2022-04-10T01:39:25.763 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:2 2022-04-10T01:39:25.763 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:39:25.763 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:39:25.764 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:2 cluster instance 2022-04-10T01:39:25.764 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:2 2022-04-10T01:39:25.764 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:39:25.764 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:39:25.764 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:39:25.765 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=2 2022-04-10T01:39:25.765 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 2 = cluster1:2 ']' 2022-04-10T01:39:25.765 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:39:25.765 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:39:25.766 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=2 2022-04-10T01:39:25.766 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:39:25.766 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:39:25.766 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:39:25.766 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:25.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:39:25.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:2 cluster1 2022-04-10T01:39:25.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:2 2022-04-10T01:39:25.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:39:25.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:39:25.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2022-04-10T01:39:25.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:2 2022-04-10T01:39:25.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:39:25.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:39:25.769 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:39:25.769 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:39:25.769 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 2 = cluster1:2 ']' 2022-04-10T01:39:25.769 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:39:25.769 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:39:25.770 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=2 2022-04-10T01:39:25.770 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=2 2022-04-10T01:39:25.770 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:39:25.770 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:39:25.771 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:25.771 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:39:25.771 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2022-04-10T01:39:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:39:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:39:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:39:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_11 2022-04-10T01:39:25.890 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:39:25.891 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:39:25.891 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:39:25.891 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:39:25.892 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:39:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:39:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:39:25.897 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:25.897 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_11 2022-04-10T01:39:25.897 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:39:25.897 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:25.898 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:25.898 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_11 2022-04-10T01:39:25.898 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:39:25.898 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:25.898 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:25.902 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:39:25.903 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:39:25.903 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_11 2022-04-10T01:39:25.903 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:39:25.903 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:39:25.903 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:25.904 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_11 2022-04-10T01:39:25.904 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:25.904 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:25.904 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:25.904 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:25.905 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:39:25.905 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:25.905 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:25.905 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:25.906 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-image_11.status 2022-04-10T01:39:25.906 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-image_11.status 2022-04-10T01:39:25.906 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:39:25.906 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_11.status 2022-04-10T01:39:25.906 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-image_11.status 2022-04-10T01:39:25.907 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-image_11.status 2022-04-10T01:39:25.907 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:39:25.907 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_11.status 2022-04-10T01:39:25.907 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror image_11 2022-04-10T01:39:25.908 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:25.908 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:25.908 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_11 2022-04-10T01:39:25.908 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:39:25.908 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:25.909 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:25.909 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:25.909 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_11 2022-04-10T01:39:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 149 2022-04-10T01:39:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_11 /tmp/tmp.rgwz74WspX/cluster2-mirror-image_11.status 2022-04-10T01:39:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_11 2022-04-10T01:39:26.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_11.status 2022-04-10T01:39:26.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:39:26.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:26.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:26.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:26.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_11 --format xml 2022-04-10T01:39:26.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:39:26.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:39:26.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=149 2022-04-10T01:39:26.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:39:26.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:39:26.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:39:26.887 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_11.status 2022-04-10T01:39:26.887 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:26.888 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:26.888 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_11 2022-04-10T01:39:26.888 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_11.status 2022-04-10T01:39:26.888 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2022-04-10T01:39:26.890 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:39:26.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:39:26.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:39:26.935 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:39:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:39:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:39:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:39:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 149 = '' 2022-04-10T01:39:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:39:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:39:27.338 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_11.status 2022-04-10T01:39:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_11 2022-04-10T01:39:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_11.status 2022-04-10T01:39:27.340 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:27.340 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:27.340 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:27.340 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:27.341 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2022-04-10T01:39:27.341 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:39:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:39:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:39:27.388 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:39:27.388 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:39:27.388 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:39:27.389 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:39:27.390 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 149 = '' 2022-04-10T01:39:27.390 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:39:27.390 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:39:28.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_11.status 2022-04-10T01:39:28.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_11 2022-04-10T01:39:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_11.status 2022-04-10T01:39:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:28.193 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:28.193 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:28.193 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2022-04-10T01:39:28.193 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:39:28.235 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:39:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=149 2022-04-10T01:39:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 149 = 149 2022-04-10T01:39:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror image_11 snap 2022-04-10T01:39:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:28.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_11 2022-04-10T01:39:28.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:39:28.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:39:28.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:28.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:28.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:28.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:28.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:39:28.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:39:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/image_11@snap 2022-04-10T01:39:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_11': 2022-04-10T01:39:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:39:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:39:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 2 2022-04-10T01:39:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 10e4adda10ae 2022-04-10T01:39:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.10e4adda10ae 2022-04-10T01:39:29.486 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:39:29.486 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:39:29.486 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:39:29.486 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:39:29.487 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:29:57 2022 2022-04-10T01:39:29.487 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:29:57 2022 2022-04-10T01:39:29.487 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:29:57 2022 2022-04-10T01:39:29.487 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:39:29.487 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:39:29.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:39:29.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: ff2a6efd-2c82-4070-8950-c9c7f1385660 2022-04-10T01:39:29.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:39:29.490 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:29.490 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror image_11 snap 2022-04-10T01:39:29.490 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:29.491 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_11 2022-04-10T01:39:29.491 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:39:29.491 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:39:29.491 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_11.export 2022-04-10T01:39:29.492 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_11.export 2022-04-10T01:39:29.492 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_11.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_11.export 2022-04-10T01:39:29.492 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export image_11@snap /tmp/tmp.rgwz74WspX/cluster2-mirror-image_11.export 2022-04-10T01:39:29.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:39:29.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export image_11@snap /tmp/tmp.rgwz74WspX/cluster1-mirror-image_11.export 2022-04-10T01:39:29.914 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:39:29.917 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-image_11.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_11.export 2022-04-10T01:39:29.982 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_11.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_11.export 2022-04-10T01:39:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:39:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_12 2022-04-10T01:39:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror image_12 snap 2022-04-10T01:39:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_12 2022-04-10T01:39:30.022 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:39:30.022 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create image_12@snap --debug-rbd=20 --debug-journaler=20 2022-04-10T01:39:31.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror image_12 2022-04-10T01:39:31.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:31.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_12 2022-04-10T01:39:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:31.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:31.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror image_12 started 2022-04-10T01:39:31.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:31.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:31.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_12 2022-04-10T01:39:31.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:39:31.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:39:31.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:31.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:31.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:31.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:31.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:39:31.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:39:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror image_12 started 2022-04-10T01:39:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_12 2022-04-10T01:39:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:39:32.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:39:32.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:39:32.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:32.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:32.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:32.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:32.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_12 2022-04-10T01:39:32.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:39:32.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:39:32.658 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:39:32.658 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:39:32.658 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:39:32.659 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:39:32.659 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:39:32.659 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:39:32.659 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:32.659 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:39:32.660 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:39:32.660 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:39:32.660 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:39:32.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:39:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:39:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2022-04-10T01:39:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:39:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:39:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:39:32.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:39:32.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:39:32.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:39:32.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:39:32.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:39:32.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:39:32.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:39:32.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:39:32.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:39:32.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:39:32.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:39:32.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:39:32.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:39:32.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:32.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:39:32.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:39:32.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:39:32.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:39:32.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:39:32.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:39:32.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:39:32.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:39:32.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:39:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:39:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:39:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:39:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:39:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:39:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:39:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:39:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:39:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:39:32.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:32.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:39:32.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:39:32.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:39:32.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:39:32.677 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:39:32.677 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2022-04-10T01:39:32.789 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2022-04-10T01:39:32.789 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:39:32.790 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:39:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:39:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_12 2022-04-10T01:39:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:1 2022-04-10T01:39:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:39:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:39:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:1 cluster instance 2022-04-10T01:39:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:1 2022-04-10T01:39:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:39:32.796 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:39:32.796 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:39:32.796 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=1 2022-04-10T01:39:32.796 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 1 = cluster1:1 ']' 2022-04-10T01:39:32.796 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:39:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:39:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=1 2022-04-10T01:39:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:39:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:39:32.798 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:39:32.798 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:32.798 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:39:32.798 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:1 cluster1 2022-04-10T01:39:32.799 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:1 2022-04-10T01:39:32.799 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:39:32.799 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:39:32.799 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2022-04-10T01:39:32.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:1 2022-04-10T01:39:32.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:39:32.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:39:32.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:39:32.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:39:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 1 = cluster1:1 ']' 2022-04-10T01:39:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:39:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:39:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=1 2022-04-10T01:39:32.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=1 2022-04-10T01:39:32.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:39:32.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:39:32.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:32.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:39:32.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2022-04-10T01:39:32.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:39:32.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:39:32.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:39:32.809 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_12 2022-04-10T01:39:32.921 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:39:32.922 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:39:32.922 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:39:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:39:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:39:32.927 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:39:32.928 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:39:32.928 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:32.928 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_12 2022-04-10T01:39:32.928 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:39:32.928 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:32.929 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:32.929 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_12 2022-04-10T01:39:32.929 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:39:32.929 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:32.930 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:32.930 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:32.930 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:39:32.930 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:39:32.930 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_12 2022-04-10T01:39:32.931 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:39:32.931 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:39:32.931 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:32.931 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_12 2022-04-10T01:39:32.932 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:32.932 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:32.932 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:32.932 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:32.932 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:39:32.933 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:32.933 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:32.933 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:32.933 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-image_12.status 2022-04-10T01:39:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-image_12.status 2022-04-10T01:39:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:39:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_12.status 2022-04-10T01:39:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-image_12.status 2022-04-10T01:39:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-image_12.status 2022-04-10T01:39:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:39:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_12.status 2022-04-10T01:39:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror image_12 2022-04-10T01:39:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_12 2022-04-10T01:39:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:39:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_12 2022-04-10T01:39:33.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 152 2022-04-10T01:39:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_12 /tmp/tmp.rgwz74WspX/cluster2-mirror-image_12.status 2022-04-10T01:39:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_12 2022-04-10T01:39:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_12.status 2022-04-10T01:39:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:39:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_12 --format xml 2022-04-10T01:39:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:39:33.712 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:39:33.714 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=152 2022-04-10T01:39:33.714 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:39:33.715 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:39:33.715 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:39:33.916 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_12.status 2022-04-10T01:39:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_12 2022-04-10T01:39:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_12.status 2022-04-10T01:39:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2022-04-10T01:39:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:39:33.967 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:39:33.967 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:39:33.967 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:39:33.968 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:39:33.968 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:39:33.968 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:39:33.968 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 152 = '' 2022-04-10T01:39:33.968 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:39:33.969 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:39:34.369 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_12.status 2022-04-10T01:39:34.369 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:34.369 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:34.370 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_12 2022-04-10T01:39:34.370 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_12.status 2022-04-10T01:39:34.371 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:34.371 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:34.371 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:34.372 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:34.372 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2022-04-10T01:39:34.372 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:39:34.416 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:39:34.417 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:39:34.418 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:39:34.419 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:39:34.419 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:39:34.420 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:39:34.420 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 152 = '' 2022-04-10T01:39:34.421 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:39:34.421 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:39:35.220 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_12.status 2022-04-10T01:39:35.220 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:35.220 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:35.221 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_12 2022-04-10T01:39:35.221 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_12.status 2022-04-10T01:39:35.221 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:35.221 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:35.222 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:35.222 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:35.222 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2022-04-10T01:39:35.222 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:39:35.263 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:39:35.265 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=152 2022-04-10T01:39:35.266 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 152 = 152 2022-04-10T01:39:35.266 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:35.266 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror image_12 snap 2022-04-10T01:39:35.266 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:35.267 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:35.267 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_12 2022-04-10T01:39:35.267 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:39:35.267 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:39:35.267 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:35.268 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:35.268 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:35.268 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:35.268 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:39:35.268 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:39:36.266 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/image_12@snap 2022-04-10T01:39:36.309 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_12': 2022-04-10T01:39:36.309 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:39:36.309 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:39:36.309 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 2 2022-04-10T01:39:36.310 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 12a75bb4edc6 2022-04-10T01:39:36.310 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.12a75bb4edc6 2022-04-10T01:39:36.310 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:39:36.310 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:39:36.311 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:39:36.311 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:39:36.311 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:30:07 2022 2022-04-10T01:39:36.311 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:30:07 2022 2022-04-10T01:39:36.312 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:30:07 2022 2022-04-10T01:39:36.312 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:39:36.312 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:39:36.312 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:39:36.312 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 672aa5fe-c483-4bea-94d8-2b504c745764 2022-04-10T01:39:36.313 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:39:36.315 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:36.316 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror image_12 snap 2022-04-10T01:39:36.316 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:36.317 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_12 2022-04-10T01:39:36.317 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:39:36.317 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:39:36.318 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_12.export 2022-04-10T01:39:36.318 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_12.export 2022-04-10T01:39:36.319 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_12.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_12.export 2022-04-10T01:39:36.319 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export image_12@snap /tmp/tmp.rgwz74WspX/cluster2-mirror-image_12.export 2022-04-10T01:39:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:39:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export image_12@snap /tmp/tmp.rgwz74WspX/cluster1-mirror-image_12.export 2022-04-10T01:39:36.750 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:39:36.753 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-image_12.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_12.export 2022-04-10T01:39:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_12.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_12.export 2022-04-10T01:39:36.854 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:36.855 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:39:36.855 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_13 2022-04-10T01:39:36.855 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror image_13 snap 2022-04-10T01:39:36.855 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:36.856 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:36.856 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_13 2022-04-10T01:39:36.856 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:39:36.856 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create image_13@snap --debug-rbd=20 --debug-journaler=20 2022-04-10T01:39:38.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror image_13 2022-04-10T01:39:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_13 2022-04-10T01:39:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror image_13 started 2022-04-10T01:39:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:38.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:38.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_13 2022-04-10T01:39:38.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:39:38.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:39:38.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:39:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:39:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror image_13 started 2022-04-10T01:39:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_13 2022-04-10T01:39:39.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:39:39.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:39:39.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:39:39.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:39.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:39.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:39.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_13 2022-04-10T01:39:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:39:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:39:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:39:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:39:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:39:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:39:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:39:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:39:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:39:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:39:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:39:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:39:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:39:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:39:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_13 2022-04-10T01:39:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:39:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:39:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:39:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:39:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:39:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:39:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:39:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:39:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:39:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:39:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:39:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:39:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:39:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:39:39.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:39:39.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:39:39.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:39.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:39:39.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:39:39.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:39:39.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:39:39.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:39:39.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:39:39.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:39:39.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:39:39.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:39:39.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:39:39.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:39:39.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:39:39.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:39:39.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:39:39.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:39:39.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:39:39.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:39:39.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:39:39.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:39.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:39:39.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:39:39.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:39:39.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:39:39.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:39:39.698 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_13 2022-04-10T01:39:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2022-04-10T01:39:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:39:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:39:39.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:39:39.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_13 2022-04-10T01:39:39.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:1 2022-04-10T01:39:39.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:39:39.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:39:39.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:1 cluster instance 2022-04-10T01:39:39.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:1 2022-04-10T01:39:39.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:39:39.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:39:39.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:39:39.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=1 2022-04-10T01:39:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 1 = cluster1:1 ']' 2022-04-10T01:39:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:39:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:39:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=1 2022-04-10T01:39:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:39:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:39:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:39:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:39:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:1 cluster1 2022-04-10T01:39:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:1 2022-04-10T01:39:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:39:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:39:39.812 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2022-04-10T01:39:39.812 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:1 2022-04-10T01:39:39.812 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:39:39.812 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:39:39.813 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:39:39.813 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:39:39.813 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 1 = cluster1:1 ']' 2022-04-10T01:39:39.813 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:39:39.813 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:39:39.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=1 2022-04-10T01:39:39.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=1 2022-04-10T01:39:39.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:39:39.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:39:39.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:39.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:39:39.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2022-04-10T01:39:39.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:39:39.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:39:39.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:39:39.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_13 2022-04-10T01:39:39.925 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2022-04-10T01:39:39.925 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:39:39.926 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:39:39.929 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:39:39.930 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_13 2022-04-10T01:39:39.930 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:2 2022-04-10T01:39:39.930 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:39:39.930 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:39:39.931 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:2 cluster instance 2022-04-10T01:39:39.931 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:2 2022-04-10T01:39:39.931 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:39:39.931 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:39:39.932 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:39:39.932 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=2 2022-04-10T01:39:39.932 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 2 = cluster1:2 ']' 2022-04-10T01:39:39.932 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:39:39.932 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:39:39.933 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=2 2022-04-10T01:39:39.933 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:39:39.933 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:39:39.933 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:39:39.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:39.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:39:39.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:2 cluster1 2022-04-10T01:39:39.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:2 2022-04-10T01:39:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:39:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:39:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2022-04-10T01:39:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:2 2022-04-10T01:39:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:39:39.936 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:39:39.936 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:39:39.936 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:39:39.936 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 2 = cluster1:2 ']' 2022-04-10T01:39:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:39:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:39:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=2 2022-04-10T01:39:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=2 2022-04-10T01:39:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:39:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:39:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:39:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2022-04-10T01:39:39.944 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:39:39.944 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:39:39.944 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:39:39.944 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_13 2022-04-10T01:39:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2022-04-10T01:39:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:39:40.051 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:39:40.055 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:39:40.055 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_13 2022-04-10T01:39:40.055 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:3 2022-04-10T01:39:40.056 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:39:40.056 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:39:40.056 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:3 cluster instance 2022-04-10T01:39:40.056 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:3 2022-04-10T01:39:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:39:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:39:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:39:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=3 2022-04-10T01:39:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 3 = cluster1:3 ']' 2022-04-10T01:39:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:39:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:39:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=3 2022-04-10T01:39:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=3 2022-04-10T01:39:40.059 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:39:40.059 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:39:40.059 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:40.059 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:39:40.060 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:3 cluster1 2022-04-10T01:39:40.060 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:3 2022-04-10T01:39:40.060 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:39:40.060 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:39:40.061 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2022-04-10T01:39:40.061 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:3 2022-04-10T01:39:40.061 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:39:40.061 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:39:40.061 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:39:40.062 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=3 2022-04-10T01:39:40.062 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 3 = cluster1:3 ']' 2022-04-10T01:39:40.062 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:39:40.062 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:39:40.063 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=3 2022-04-10T01:39:40.063 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=3 2022-04-10T01:39:40.063 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:39:40.063 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:39:40.064 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:40.064 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:39:40.064 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2022-04-10T01:39:40.068 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:39:40.069 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:39:40.069 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:39:40.069 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_13 2022-04-10T01:39:40.182 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:39:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:39:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:39:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:39:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:39:40.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:39:40.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:39:40.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:40.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_13 2022-04-10T01:39:40.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:39:40.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:40.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:40.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_13 2022-04-10T01:39:40.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:39:40.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:40.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:40.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:39:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:39:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_13 2022-04-10T01:39:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:39:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:39:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_13 2022-04-10T01:39:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:40.194 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:40.194 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:40.194 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:39:40.194 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:40.195 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:40.195 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:40.195 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-image_13.status 2022-04-10T01:39:40.195 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-image_13.status 2022-04-10T01:39:40.195 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:39:40.196 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_13.status 2022-04-10T01:39:40.196 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-image_13.status 2022-04-10T01:39:40.196 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-image_13.status 2022-04-10T01:39:40.196 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:39:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_13.status 2022-04-10T01:39:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror image_13 2022-04-10T01:39:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_13 2022-04-10T01:39:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:39:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:40.199 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_13 2022-04-10T01:39:40.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 155 2022-04-10T01:39:40.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_13 /tmp/tmp.rgwz74WspX/cluster2-mirror-image_13.status 2022-04-10T01:39:40.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:40.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:40.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_13 2022-04-10T01:39:40.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_13.status 2022-04-10T01:39:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:39:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:40.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:40.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_13 --format xml 2022-04-10T01:39:40.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:39:40.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:39:40.723 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=155 2022-04-10T01:39:40.723 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:39:40.723 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:39:40.723 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:39:40.924 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_13.status 2022-04-10T01:39:40.924 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:40.925 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:40.925 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_13 2022-04-10T01:39:40.925 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_13.status 2022-04-10T01:39:40.925 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:40.926 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:40.926 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:40.926 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:40.926 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2022-04-10T01:39:40.926 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:39:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:39:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:39:40.972 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:39:40.972 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:39:40.973 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:39:40.974 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:39:40.974 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 155 = '' 2022-04-10T01:39:40.974 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:39:40.974 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:39:41.375 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_13.status 2022-04-10T01:39:41.376 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:41.376 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:41.376 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_13 2022-04-10T01:39:41.376 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_13.status 2022-04-10T01:39:41.377 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:41.377 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:41.377 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:41.377 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:41.378 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2022-04-10T01:39:41.378 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:39:41.423 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:39:41.423 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:39:41.424 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:39:41.424 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:39:41.425 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:39:41.425 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:39:41.426 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 155 = '' 2022-04-10T01:39:41.426 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:39:41.426 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:39:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_13.status 2022-04-10T01:39:42.228 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:42.228 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:42.228 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_13 2022-04-10T01:39:42.229 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_13.status 2022-04-10T01:39:42.229 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:42.229 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:42.229 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:42.229 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:42.230 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2022-04-10T01:39:42.230 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:39:42.269 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:39:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=155 2022-04-10T01:39:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 155 = 155 2022-04-10T01:39:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror image_13 snap 2022-04-10T01:39:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:42.273 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:42.273 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_13 2022-04-10T01:39:42.273 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:39:42.273 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:39:42.273 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:39:42.275 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:39:43.272 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/image_13@snap 2022-04-10T01:39:43.316 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_13': 2022-04-10T01:39:43.316 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:39:43.316 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:39:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 3 2022-04-10T01:39:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 10e4a9f38464 2022-04-10T01:39:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.10e4a9f38464 2022-04-10T01:39:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:39:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, operations, non-primary 2022-04-10T01:39:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: snap-trash 2022-04-10T01:39:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:39:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:30:20 2022 2022-04-10T01:39:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:30:20 2022 2022-04-10T01:39:43.319 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:30:20 2022 2022-04-10T01:39:43.319 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:39:43.319 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:39:43.319 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:39:43.320 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: b7fe287d-2790-4653-8a15-70f6e8da8e9f 2022-04-10T01:39:43.320 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:39:43.321 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:43.321 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror image_13 snap 2022-04-10T01:39:43.322 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:43.322 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_13 2022-04-10T01:39:43.322 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:39:43.322 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:39:43.323 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_13.export 2022-04-10T01:39:43.323 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_13.export 2022-04-10T01:39:43.323 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_13.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_13.export 2022-04-10T01:39:43.323 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export image_13@snap /tmp/tmp.rgwz74WspX/cluster2-mirror-image_13.export 2022-04-10T01:39:43.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:39:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export image_13@snap /tmp/tmp.rgwz74WspX/cluster1-mirror-image_13.export 2022-04-10T01:39:43.764 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:39:43.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-image_13.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_13.export 2022-04-10T01:39:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_13.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_13.export 2022-04-10T01:39:43.869 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:43.869 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:39:43.870 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_14 2022-04-10T01:39:43.870 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror image_14 snap 2022-04-10T01:39:43.870 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:43.870 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:43.871 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_14 2022-04-10T01:39:43.871 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:39:43.871 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create image_14@snap --debug-rbd=20 --debug-journaler=20 2022-04-10T01:39:45.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror image_14 2022-04-10T01:39:45.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:45.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:45.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_14 2022-04-10T01:39:45.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:45.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:45.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:45.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:45.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror image_14 started 2022-04-10T01:39:45.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:45.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:45.500 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_14 2022-04-10T01:39:45.500 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:39:45.500 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:39:45.500 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:45.501 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:45.501 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:45.501 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:45.501 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:39:45.502 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:39:46.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror image_14 started 2022-04-10T01:39:46.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:46.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:46.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_14 2022-04-10T01:39:46.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:39:46.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:39:46.500 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:39:46.500 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:46.500 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:46.500 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:46.500 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_14 2022-04-10T01:39:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:39:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:39:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:39:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:39:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:39:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:39:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:39:46.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:39:46.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:46.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:39:46.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:39:46.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:39:46.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:39:46.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:39:46.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:39:46.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_14 2022-04-10T01:39:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:39:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:39:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:39:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:39:46.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:39:46.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:39:46.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:39:46.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:39:46.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:39:46.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:39:46.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:39:46.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:39:46.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:39:46.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:39:46.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:39:46.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:39:46.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:46.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:39:46.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:39:46.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:39:46.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:39:46.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:39:46.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:39:46.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:39:46.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:39:46.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:39:46.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:39:46.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:39:46.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:39:46.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:39:46.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:39:46.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:39:46.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:39:46.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:39:46.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:39:46.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:46.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:39:46.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:39:46.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:39:46.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:39:46.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:39:46.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_14 2022-04-10T01:39:46.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2022-04-10T01:39:46.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:39:46.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:39:46.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:39:46.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_14 2022-04-10T01:39:46.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:1 2022-04-10T01:39:46.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:39:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:39:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:1 cluster instance 2022-04-10T01:39:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:1 2022-04-10T01:39:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:39:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:39:46.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:39:46.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=1 2022-04-10T01:39:46.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 1 = cluster1:1 ']' 2022-04-10T01:39:46.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:39:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:39:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=1 2022-04-10T01:39:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:39:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:39:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:39:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:39:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:1 cluster1 2022-04-10T01:39:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:1 2022-04-10T01:39:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:39:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:39:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2022-04-10T01:39:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:1 2022-04-10T01:39:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:39:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:39:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:39:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:39:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 1 = cluster1:1 ']' 2022-04-10T01:39:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:39:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:39:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=1 2022-04-10T01:39:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=1 2022-04-10T01:39:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:39:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:39:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:39:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2022-04-10T01:39:46.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:39:46.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:39:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:39:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_14 2022-04-10T01:39:46.757 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2022-04-10T01:39:46.757 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:39:46.758 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:39:46.761 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:39:46.762 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_14 2022-04-10T01:39:46.762 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:2 2022-04-10T01:39:46.762 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:39:46.762 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:39:46.762 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:2 cluster instance 2022-04-10T01:39:46.763 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:2 2022-04-10T01:39:46.763 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:39:46.763 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:39:46.763 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:39:46.764 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=2 2022-04-10T01:39:46.764 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 2 = cluster1:2 ']' 2022-04-10T01:39:46.764 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:39:46.764 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:39:46.764 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=2 2022-04-10T01:39:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:39:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:39:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:39:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:39:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:2 cluster1 2022-04-10T01:39:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:2 2022-04-10T01:39:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:39:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:39:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2022-04-10T01:39:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:2 2022-04-10T01:39:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:39:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:39:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:39:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:39:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 2 = cluster1:2 ']' 2022-04-10T01:39:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:39:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:39:46.769 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=2 2022-04-10T01:39:46.769 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=2 2022-04-10T01:39:46.769 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:39:46.769 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:39:46.769 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:46.770 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:39:46.770 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2022-04-10T01:39:46.775 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:39:46.776 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:39:46.776 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:39:46.776 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_14 2022-04-10T01:39:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2022-04-10T01:39:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:39:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:39:46.885 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:39:46.885 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_14 2022-04-10T01:39:46.885 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:3 2022-04-10T01:39:46.885 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:39:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:39:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:3 cluster instance 2022-04-10T01:39:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:3 2022-04-10T01:39:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:39:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:39:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:39:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=3 2022-04-10T01:39:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 3 = cluster1:3 ']' 2022-04-10T01:39:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:39:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:39:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=3 2022-04-10T01:39:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=3 2022-04-10T01:39:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:39:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:39:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:46.889 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:39:46.889 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:3 cluster1 2022-04-10T01:39:46.889 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:3 2022-04-10T01:39:46.889 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:39:46.890 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:39:46.890 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2022-04-10T01:39:46.890 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:3 2022-04-10T01:39:46.890 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:39:46.890 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:39:46.891 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:39:46.891 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=3 2022-04-10T01:39:46.891 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 3 = cluster1:3 ']' 2022-04-10T01:39:46.891 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:39:46.891 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:39:46.892 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=3 2022-04-10T01:39:46.892 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=3 2022-04-10T01:39:46.892 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:39:46.892 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:39:46.892 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:46.893 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:39:46.893 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2022-04-10T01:39:46.899 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:39:46.900 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:39:46.900 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:39:46.900 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_14 2022-04-10T01:39:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:39:47.014 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:39:47.014 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:39:47.015 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:39:47.015 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:39:47.018 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:39:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:39:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_14 2022-04-10T01:39:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:39:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_14 2022-04-10T01:39:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:39:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:39:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:39:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_14 2022-04-10T01:39:47.022 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:39:47.022 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:39:47.022 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:47.022 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_14 2022-04-10T01:39:47.022 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:39:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:47.024 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:47.024 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:47.024 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-image_14.status 2022-04-10T01:39:47.024 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-image_14.status 2022-04-10T01:39:47.024 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:39:47.025 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_14.status 2022-04-10T01:39:47.025 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-image_14.status 2022-04-10T01:39:47.025 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-image_14.status 2022-04-10T01:39:47.025 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:39:47.027 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_14.status 2022-04-10T01:39:47.027 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror image_14 2022-04-10T01:39:47.027 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:47.027 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:47.027 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_14 2022-04-10T01:39:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:39:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_14 2022-04-10T01:39:47.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 158 2022-04-10T01:39:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_14 /tmp/tmp.rgwz74WspX/cluster2-mirror-image_14.status 2022-04-10T01:39:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:47.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_14 2022-04-10T01:39:47.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_14.status 2022-04-10T01:39:47.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:39:47.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:47.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:47.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:47.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_14 --format xml 2022-04-10T01:39:47.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:39:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:39:47.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=158 2022-04-10T01:39:47.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:39:47.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:39:47.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:39:47.752 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_14.status 2022-04-10T01:39:47.753 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:47.753 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:47.753 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_14 2022-04-10T01:39:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_14.status 2022-04-10T01:39:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:47.755 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:47.755 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2022-04-10T01:39:47.755 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:39:47.798 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:39:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:39:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:39:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:39:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:39:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:39:47.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 158 = '' 2022-04-10T01:39:47.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:39:47.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:39:48.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_14.status 2022-04-10T01:39:48.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:48.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:48.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_14 2022-04-10T01:39:48.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_14.status 2022-04-10T01:39:48.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:48.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:48.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:48.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:48.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2022-04-10T01:39:48.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:39:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:39:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:39:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:39:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:39:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:39:48.254 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:39:48.254 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 158 = '' 2022-04-10T01:39:48.254 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:39:48.255 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:39:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_14.status 2022-04-10T01:39:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_14 2022-04-10T01:39:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_14.status 2022-04-10T01:39:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:49.058 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:49.058 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2022-04-10T01:39:49.058 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:39:49.098 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:39:49.100 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=158 2022-04-10T01:39:49.100 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 158 = 158 2022-04-10T01:39:49.101 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:49.101 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror image_14 snap 2022-04-10T01:39:49.101 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:49.101 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_14 2022-04-10T01:39:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:39:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:39:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:49.103 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:49.103 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:49.103 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:39:49.103 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:39:50.101 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/image_14@snap 2022-04-10T01:39:50.144 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_14': 2022-04-10T01:39:50.145 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:39:50.145 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:39:50.145 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 2 2022-04-10T01:39:50.145 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 11334183c9c7 2022-04-10T01:39:50.146 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.11334183c9c7 2022-04-10T01:39:50.146 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:39:50.146 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:39:50.146 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:39:50.147 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:39:50.147 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:30:30 2022 2022-04-10T01:39:50.147 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:30:30 2022 2022-04-10T01:39:50.147 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:30:30 2022 2022-04-10T01:39:50.147 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:39:50.148 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:39:50.148 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:39:50.148 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 9daee277-0f48-4a3a-be63-d5fc81096bc5 2022-04-10T01:39:50.148 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:39:50.150 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:50.150 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror image_14 snap 2022-04-10T01:39:50.150 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:50.151 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_14 2022-04-10T01:39:50.151 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:39:50.151 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:39:50.151 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_14.export 2022-04-10T01:39:50.152 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_14.export 2022-04-10T01:39:50.152 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_14.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_14.export 2022-04-10T01:39:50.152 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export image_14@snap /tmp/tmp.rgwz74WspX/cluster2-mirror-image_14.export 2022-04-10T01:39:50.338 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:39:50.341 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export image_14@snap /tmp/tmp.rgwz74WspX/cluster1-mirror-image_14.export 2022-04-10T01:39:50.587 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:39:50.590 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-image_14.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_14.export 2022-04-10T01:39:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_14.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_14.export 2022-04-10T01:39:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:39:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_15 2022-04-10T01:39:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror image_15 snap 2022-04-10T01:39:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_15 2022-04-10T01:39:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:39:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create image_15@snap --debug-rbd=20 --debug-journaler=20 2022-04-10T01:39:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror image_15 2022-04-10T01:39:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_15 2022-04-10T01:39:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror image_15 started 2022-04-10T01:39:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:52.500 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_15 2022-04-10T01:39:52.500 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:39:52.500 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:39:52.500 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:52.500 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:39:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:39:53.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror image_15 started 2022-04-10T01:39:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_15 2022-04-10T01:39:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:39:53.500 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:39:53.500 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:39:53.500 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:53.500 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:53.500 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:53.501 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:53.501 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_15 2022-04-10T01:39:53.501 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:39:53.501 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:39:53.501 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:39:53.502 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:39:53.502 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:39:53.502 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:39:53.502 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:39:53.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:39:53.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:53.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:39:53.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:39:53.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:39:53.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:39:53.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:39:53.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:39:53.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_15 2022-04-10T01:39:53.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:39:53.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:39:53.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:39:53.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:39:53.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:39:53.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:39:53.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:39:53.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:39:53.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:39:53.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:39:53.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:39:53.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:39:53.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:39:53.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:39:53.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:39:53.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:39:53.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:53.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:39:53.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:39:53.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:39:53.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:39:53.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:39:53.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:39:53.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:39:53.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:39:53.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:39:53.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:39:53.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:39:53.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:39:53.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:39:53.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:39:53.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:39:53.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:39:53.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:39:53.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:39:53.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:53.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:39:53.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:39:53.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:39:53.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:39:53.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:39:53.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_15 2022-04-10T01:39:53.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:39:53.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:39:53.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:39:53.647 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:39:53.647 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:39:53.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:39:53.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:39:53.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:53.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_15 2022-04-10T01:39:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:39:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:53.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_15 2022-04-10T01:39:53.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:39:53.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:53.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:53.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:53.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:39:53.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:39:53.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_15 2022-04-10T01:39:53.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:39:53.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:39:53.658 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:53.658 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_15 2022-04-10T01:39:53.658 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:53.658 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:39:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:53.660 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:53.660 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:53.660 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-image_15.status 2022-04-10T01:39:53.661 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-image_15.status 2022-04-10T01:39:53.661 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:39:53.661 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_15.status 2022-04-10T01:39:53.662 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-image_15.status 2022-04-10T01:39:53.662 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-image_15.status 2022-04-10T01:39:53.662 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:39:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_15.status 2022-04-10T01:39:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror image_15 2022-04-10T01:39:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:53.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:53.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_15 2022-04-10T01:39:53.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:39:53.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_15 2022-04-10T01:39:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 161 2022-04-10T01:39:54.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_15 /tmp/tmp.rgwz74WspX/cluster2-mirror-image_15.status 2022-04-10T01:39:54.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:54.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:54.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_15 2022-04-10T01:39:54.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_15.status 2022-04-10T01:39:54.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:39:54.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:54.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:54.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:54.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_15 --format xml 2022-04-10T01:39:54.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:39:54.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:39:54.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=161 2022-04-10T01:39:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:39:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:39:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:39:54.765 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_15.status 2022-04-10T01:39:54.766 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:54.766 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:54.766 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_15 2022-04-10T01:39:54.766 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_15.status 2022-04-10T01:39:54.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:54.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:54.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:54.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:54.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2022-04-10T01:39:54.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:39:54.811 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:39:54.811 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:39:54.812 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:39:54.812 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:39:54.813 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:39:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:39:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 161 = '' 2022-04-10T01:39:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:39:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:39:55.215 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_15.status 2022-04-10T01:39:55.215 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:55.215 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:55.216 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_15 2022-04-10T01:39:55.216 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_15.status 2022-04-10T01:39:55.216 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:55.216 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:55.216 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2022-04-10T01:39:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:39:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:39:55.265 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:39:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:39:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:39:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:39:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:39:55.268 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 161 = '' 2022-04-10T01:39:55.268 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:39:55.268 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:39:56.069 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_15.status 2022-04-10T01:39:56.069 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:56.069 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:56.069 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_15 2022-04-10T01:39:56.070 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_15.status 2022-04-10T01:39:56.070 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:56.070 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:56.070 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:56.071 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:56.071 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2022-04-10T01:39:56.071 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:39:56.111 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:39:56.114 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=161 2022-04-10T01:39:56.114 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 161 = 161 2022-04-10T01:39:56.114 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:56.114 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror image_15 snap 2022-04-10T01:39:56.115 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:56.115 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:56.115 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_15 2022-04-10T01:39:56.115 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:39:56.116 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:39:56.116 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:56.116 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:56.116 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:56.116 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:56.117 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:39:56.117 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:39:57.114 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/image_15@snap 2022-04-10T01:39:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_15': 2022-04-10T01:39:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:39:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:39:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 2 2022-04-10T01:39:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 118d1ade7f20 2022-04-10T01:39:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.118d1ade7f20 2022-04-10T01:39:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:39:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:39:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:39:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:39:57.160 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:30:40 2022 2022-04-10T01:39:57.160 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:30:40 2022 2022-04-10T01:39:57.160 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:30:40 2022 2022-04-10T01:39:57.160 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:39:57.161 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:39:57.161 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:39:57.161 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 1de7d6c4-ab1c-49a3-8053-b58d65c37430 2022-04-10T01:39:57.161 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:39:57.163 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:57.163 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror image_15 snap 2022-04-10T01:39:57.163 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:57.163 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_15 2022-04-10T01:39:57.164 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:39:57.164 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:39:57.164 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_15.export 2022-04-10T01:39:57.164 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_15.export 2022-04-10T01:39:57.165 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_15.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_15.export 2022-04-10T01:39:57.165 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export image_15@snap /tmp/tmp.rgwz74WspX/cluster2-mirror-image_15.export 2022-04-10T01:39:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:39:57.353 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export image_15@snap /tmp/tmp.rgwz74WspX/cluster1-mirror-image_15.export 2022-04-10T01:39:57.590 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:39:57.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-image_15.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_15.export 2022-04-10T01:39:57.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_15.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_15.export 2022-04-10T01:39:57.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:57.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:39:57.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_16 2022-04-10T01:39:57.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror image_16 snap 2022-04-10T01:39:57.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:39:57.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:57.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_16 2022-04-10T01:39:57.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:39:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create image_16@snap --debug-rbd=20 --debug-journaler=20 2022-04-10T01:39:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror image_16 2022-04-10T01:39:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_16 2022-04-10T01:39:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror image_16 started 2022-04-10T01:39:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:39:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_16 2022-04-10T01:39:59.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:39:59.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:39:59.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:39:59.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:39:59.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:39:59.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:39:59.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:39:59.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:40:00.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror image_16 started 2022-04-10T01:40:00.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_16 2022-04-10T01:40:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:40:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:40:00.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:40:00.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:00.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:00.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:00.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:00.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_16 2022-04-10T01:40:00.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:40:00.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:40:00.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:40:00.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:40:00.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:40:00.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:40:00.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:40:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:40:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:40:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:40:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:40:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:40:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:40:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:40:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_16 2022-04-10T01:40:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:40:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:40:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:40:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:40:00.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:40:00.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:40:00.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:40:00.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:40:00.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:40:00.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:40:00.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:40:00.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:40:00.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:40:00.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:40:00.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:40:00.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:40:00.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:00.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:40:00.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:40:00.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:40:00.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:40:00.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:40:00.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:40:00.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:40:00.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:40:00.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:40:00.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:40:00.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:40:00.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:40:00.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:40:00.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:40:00.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:40:00.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:40:00.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:40:00.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:40:00.528 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:00.528 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:40:00.528 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:40:00.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:40:00.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:40:00.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:40:00.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_16 2022-04-10T01:40:00.649 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2022-04-10T01:40:00.650 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:40:00.650 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:40:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:40:00.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_16 2022-04-10T01:40:00.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:1 2022-04-10T01:40:00.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:40:00.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:40:00.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:1 cluster instance 2022-04-10T01:40:00.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:1 2022-04-10T01:40:00.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:40:00.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:40:00.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:40:00.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=1 2022-04-10T01:40:00.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 1 = cluster1:1 ']' 2022-04-10T01:40:00.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:40:00.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:40:00.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=1 2022-04-10T01:40:00.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:40:00.658 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:40:00.658 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:40:00.658 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:00.658 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:40:00.658 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:1 cluster1 2022-04-10T01:40:00.659 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:1 2022-04-10T01:40:00.659 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:40:00.659 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:40:00.659 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2022-04-10T01:40:00.660 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:1 2022-04-10T01:40:00.660 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:40:00.660 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:40:00.660 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:40:00.661 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:40:00.661 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 1 = cluster1:1 ']' 2022-04-10T01:40:00.661 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:40:00.661 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:40:00.661 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=1 2022-04-10T01:40:00.662 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=1 2022-04-10T01:40:00.662 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:40:00.662 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:40:00.662 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:00.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:40:00.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2022-04-10T01:40:00.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:40:00.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:40:00.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:40:00.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_16 2022-04-10T01:40:00.773 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2022-04-10T01:40:00.774 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:40:00.774 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:40:00.778 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:40:00.778 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_16 2022-04-10T01:40:00.778 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:2 2022-04-10T01:40:00.779 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:40:00.779 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:40:00.779 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:2 cluster instance 2022-04-10T01:40:00.779 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:2 2022-04-10T01:40:00.780 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:40:00.780 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:40:00.780 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:40:00.780 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=2 2022-04-10T01:40:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 2 = cluster1:2 ']' 2022-04-10T01:40:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:40:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:40:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=2 2022-04-10T01:40:00.782 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:40:00.782 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:40:00.782 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:40:00.782 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:00.782 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:40:00.783 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:2 cluster1 2022-04-10T01:40:00.783 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:2 2022-04-10T01:40:00.783 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:40:00.783 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:40:00.784 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2022-04-10T01:40:00.784 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:2 2022-04-10T01:40:00.784 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:40:00.784 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:40:00.784 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:40:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:40:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 2 = cluster1:2 ']' 2022-04-10T01:40:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:40:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:40:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=2 2022-04-10T01:40:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=2 2022-04-10T01:40:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:40:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:40:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:40:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2022-04-10T01:40:00.792 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:40:00.792 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:40:00.792 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:40:00.793 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_16 2022-04-10T01:40:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:40:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:40:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:40:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:40:00.908 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:40:00.912 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:40:00.913 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:40:00.913 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:00.913 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_16 2022-04-10T01:40:00.913 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:40:00.914 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:00.914 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:00.914 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_16 2022-04-10T01:40:00.914 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:40:00.915 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:00.915 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:00.915 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:00.915 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:40:00.916 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:40:00.916 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_16 2022-04-10T01:40:00.916 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:40:00.916 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:40:00.916 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:00.917 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_16 2022-04-10T01:40:00.917 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:00.917 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:00.917 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:00.918 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:00.918 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:40:00.918 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:00.918 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:00.919 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:00.919 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-image_16.status 2022-04-10T01:40:00.919 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-image_16.status 2022-04-10T01:40:00.919 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:40:00.919 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_16.status 2022-04-10T01:40:00.920 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-image_16.status 2022-04-10T01:40:00.920 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-image_16.status 2022-04-10T01:40:00.920 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:40:00.920 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_16.status 2022-04-10T01:40:00.921 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror image_16 2022-04-10T01:40:00.921 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:00.921 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:00.921 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_16 2022-04-10T01:40:00.922 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:40:00.922 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:00.922 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:00.922 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:00.922 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_16 2022-04-10T01:40:01.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 164 2022-04-10T01:40:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_16 /tmp/tmp.rgwz74WspX/cluster2-mirror-image_16.status 2022-04-10T01:40:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_16 2022-04-10T01:40:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_16.status 2022-04-10T01:40:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:40:01.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:01.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:01.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:01.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_16 --format xml 2022-04-10T01:40:01.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:40:01.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:40:01.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=164 2022-04-10T01:40:01.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:40:01.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:40:01.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:40:01.782 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_16.status 2022-04-10T01:40:01.783 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:01.783 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:01.783 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_16 2022-04-10T01:40:01.783 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_16.status 2022-04-10T01:40:01.783 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:01.784 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:01.784 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:01.784 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:01.784 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2022-04-10T01:40:01.784 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:40:01.832 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:40:01.832 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:40:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:40:01.834 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:40:01.834 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:40:01.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:40:01.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 164 = '' 2022-04-10T01:40:01.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:40:01.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:40:02.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_16.status 2022-04-10T01:40:02.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:02.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:02.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_16 2022-04-10T01:40:02.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_16.status 2022-04-10T01:40:02.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:02.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:02.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:02.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:02.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2022-04-10T01:40:02.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:40:02.285 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:40:02.285 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:40:02.286 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:40:02.286 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:40:02.287 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:40:02.287 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:40:02.288 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 164 = '' 2022-04-10T01:40:02.288 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:40:02.288 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:40:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_16.status 2022-04-10T01:40:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_16 2022-04-10T01:40:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_16.status 2022-04-10T01:40:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:03.091 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:03.091 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:03.091 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2022-04-10T01:40:03.091 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:40:03.333 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:40:03.336 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=164 2022-04-10T01:40:03.336 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 164 = 164 2022-04-10T01:40:03.336 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:03.337 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror image_16 snap 2022-04-10T01:40:03.337 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:03.337 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:03.337 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_16 2022-04-10T01:40:03.337 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:40:03.338 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:40:03.338 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:03.338 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:03.338 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:03.339 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:03.339 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:40:03.339 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:40:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/image_16@snap 2022-04-10T01:40:04.378 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_16': 2022-04-10T01:40:04.378 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:40:04.378 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:40:04.378 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 2 2022-04-10T01:40:04.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 12a731a0d665 2022-04-10T01:40:04.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.12a731a0d665 2022-04-10T01:40:04.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:40:04.380 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:40:04.380 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:40:04.380 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:40:04.380 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:30:54 2022 2022-04-10T01:40:04.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:30:54 2022 2022-04-10T01:40:04.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:30:54 2022 2022-04-10T01:40:04.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:40:04.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:40:04.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:40:04.382 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 47fb89d8-4dfe-49df-833b-c0b0fd34dfb9 2022-04-10T01:40:04.382 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:40:04.383 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror image_16 snap 2022-04-10T01:40:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_16 2022-04-10T01:40:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:40:04.385 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:40:04.385 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_16.export 2022-04-10T01:40:04.385 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_16.export 2022-04-10T01:40:04.385 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_16.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_16.export 2022-04-10T01:40:04.386 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export image_16@snap /tmp/tmp.rgwz74WspX/cluster2-mirror-image_16.export 2022-04-10T01:40:04.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:40:04.566 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export image_16@snap /tmp/tmp.rgwz74WspX/cluster1-mirror-image_16.export 2022-04-10T01:40:04.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:40:04.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-image_16.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_16.export 2022-04-10T01:40:04.860 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7008043281785148) > weight (0.25) 2022-04-10T01:40:04.861 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2022-04-10T01:40:04.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2022-04-10T01:40:04.862 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2022-04-10T01:40:04.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi043.stderr:2022-04-10T01:40:04.860+0000 7f89f7e44700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 43731) UID: 1000 2022-04-10T01:40:04.870 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_16.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_16.export 2022-04-10T01:40:04.919 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:04.919 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:40:04.919 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_17 2022-04-10T01:40:04.919 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror image_17 snap 2022-04-10T01:40:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_17 2022-04-10T01:40:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:40:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create image_17@snap --debug-rbd=20 --debug-journaler=20 2022-04-10T01:40:05.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi043.stderr:daemon-helper: command failed with exit status 1 2022-04-10T01:40:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror image_17 2022-04-10T01:40:06.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:06.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:06.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_17 2022-04-10T01:40:06.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror image_17 started 2022-04-10T01:40:06.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:06.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:06.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_17 2022-04-10T01:40:06.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:40:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:40:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:40:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:40:07.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror image_17 started 2022-04-10T01:40:07.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:07.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:07.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_17 2022-04-10T01:40:07.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:40:07.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:40:07.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:40:07.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:07.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:07.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:07.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:07.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_17 2022-04-10T01:40:07.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:40:07.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:40:07.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:40:07.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:40:07.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:40:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:40:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:40:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:40:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:40:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:40:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:40:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:40:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:40:07.542 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:40:07.543 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_17 2022-04-10T01:40:07.543 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:40:07.543 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:40:07.543 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:40:07.543 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:40:07.544 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:40:07.544 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:40:07.544 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:40:07.544 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:40:07.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:40:07.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:40:07.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:40:07.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:40:07.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:40:07.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:40:07.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:40:07.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:40:07.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:07.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:40:07.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:40:07.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:40:07.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:40:07.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:40:07.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:40:07.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:40:07.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:40:07.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:40:07.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:40:07.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:40:07.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:40:07.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:40:07.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:40:07.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:40:07.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:40:07.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:40:07.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:40:07.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:07.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:40:07.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:40:07.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:40:07.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:40:07.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:40:07.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_17 2022-04-10T01:40:07.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:40:07.677 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:40:07.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:40:07.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:40:07.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:40:07.681 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:40:07.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:40:07.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:07.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_17 2022-04-10T01:40:07.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:40:07.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:07.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:07.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_17 2022-04-10T01:40:07.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:40:07.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:40:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:40:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_17 2022-04-10T01:40:07.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:40:07.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:40:07.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:07.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_17 2022-04-10T01:40:07.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:07.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:07.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:07.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:07.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:40:07.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:07.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:07.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:07.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-image_17.status 2022-04-10T01:40:07.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-image_17.status 2022-04-10T01:40:07.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:40:07.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_17.status 2022-04-10T01:40:07.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-image_17.status 2022-04-10T01:40:07.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-image_17.status 2022-04-10T01:40:07.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:40:07.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_17.status 2022-04-10T01:40:07.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror image_17 2022-04-10T01:40:07.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:07.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:07.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_17 2022-04-10T01:40:07.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:40:07.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:07.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:07.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:07.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_17 2022-04-10T01:40:08.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 167 2022-04-10T01:40:08.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_17 /tmp/tmp.rgwz74WspX/cluster2-mirror-image_17.status 2022-04-10T01:40:08.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:08.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:08.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_17 2022-04-10T01:40:08.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_17.status 2022-04-10T01:40:08.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:40:08.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:08.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:08.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:08.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_17 --format xml 2022-04-10T01:40:08.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:40:08.591 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:40:08.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=167 2022-04-10T01:40:08.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:40:08.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:40:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:40:08.795 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_17.status 2022-04-10T01:40:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_17 2022-04-10T01:40:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_17.status 2022-04-10T01:40:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:08.797 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:08.797 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:08.797 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:08.797 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2022-04-10T01:40:08.798 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:40:08.843 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:40:08.844 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:40:08.845 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:40:08.846 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:40:08.846 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:40:08.846 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:40:08.846 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 167 = '' 2022-04-10T01:40:08.846 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:40:08.847 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:40:09.247 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_17.status 2022-04-10T01:40:09.248 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:09.248 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:09.248 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_17 2022-04-10T01:40:09.248 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_17.status 2022-04-10T01:40:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2022-04-10T01:40:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:40:09.293 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:40:09.294 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:40:09.295 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:40:09.295 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:40:09.295 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:40:09.296 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:40:09.296 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 167 = '' 2022-04-10T01:40:09.297 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:40:09.297 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:40:10.097 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_17.status 2022-04-10T01:40:10.097 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:10.098 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:10.098 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_17 2022-04-10T01:40:10.098 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_17.status 2022-04-10T01:40:10.098 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:10.098 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:10.099 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:10.099 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:10.099 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2022-04-10T01:40:10.099 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:40:10.139 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:40:10.141 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=167 2022-04-10T01:40:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 167 = 167 2022-04-10T01:40:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror image_17 snap 2022-04-10T01:40:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_17 2022-04-10T01:40:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:40:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:40:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:40:10.145 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:40:10.176 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.38720030040352493) > weight (0.25) 2022-04-10T01:40:10.177 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2022-04-10T01:40:10.177 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2022-04-10T01:40:10.177 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2022-04-10T01:40:10.178 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi043.stderr:2022-04-10T01:40:10.176+0000 7ff4ea0b9700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 54761) UID: 1000 2022-04-10T01:40:10.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi043.stderr:daemon-helper: command failed with exit status 1 2022-04-10T01:40:11.142 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/image_17@snap 2022-04-10T01:40:11.182 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_17': 2022-04-10T01:40:11.183 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:40:11.183 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:40:11.183 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 2 2022-04-10T01:40:11.183 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 10e493af4748 2022-04-10T01:40:11.184 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.10e493af4748 2022-04-10T01:40:11.184 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:40:11.184 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:40:11.184 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:40:11.184 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:40:11.185 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:31:00 2022 2022-04-10T01:40:11.185 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:31:00 2022 2022-04-10T01:40:11.185 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:31:00 2022 2022-04-10T01:40:11.185 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:40:11.185 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:40:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:40:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 59134803-ff09-429b-9467-2cdbc3d6d23c 2022-04-10T01:40:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:40:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:11.188 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror image_17 snap 2022-04-10T01:40:11.188 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:11.188 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_17 2022-04-10T01:40:11.188 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:40:11.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:40:11.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_17.export 2022-04-10T01:40:11.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_17.export 2022-04-10T01:40:11.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_17.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_17.export 2022-04-10T01:40:11.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export image_17@snap /tmp/tmp.rgwz74WspX/cluster2-mirror-image_17.export 2022-04-10T01:40:11.368 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:40:11.371 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export image_17@snap /tmp/tmp.rgwz74WspX/cluster1-mirror-image_17.export 2022-04-10T01:40:11.602 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:40:11.606 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-image_17.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_17.export 2022-04-10T01:40:11.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_17.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_17.export 2022-04-10T01:40:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:11.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:40:11.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_18 2022-04-10T01:40:11.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror image_18 snap 2022-04-10T01:40:11.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:11.709 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:11.709 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_18 2022-04-10T01:40:11.709 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:40:11.709 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create image_18@snap --debug-rbd=20 --debug-journaler=20 2022-04-10T01:40:12.863 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2022-04-10T01:40:12.863 INFO:teuthology.orchestra.run:waiting for 600 2022-04-10T01:40:12.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-04-10T01:40:12.864 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2022-04-10T01:40:12.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2022-04-10T01:40:12.865 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2022-04-10T01:40:12.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2022-04-10T01:40:12.868 INFO:tasks.rbd_mirror_thrash:waiting for 98 secs before thrashing 2022-04-10T01:40:13.178 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2022-04-10T01:40:13.179 INFO:teuthology.orchestra.run:waiting for 600 2022-04-10T01:40:13.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-04-10T01:40:13.179 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2022-04-10T01:40:13.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2022-04-10T01:40:13.180 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2022-04-10T01:40:13.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2022-04-10T01:40:13.183 INFO:tasks.rbd_mirror_thrash:waiting for 60 secs before thrashing 2022-04-10T01:40:13.544 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror image_18 2022-04-10T01:40:13.544 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:13.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:13.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_18 2022-04-10T01:40:13.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:13.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:13.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:13.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:13.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror image_18 started 2022-04-10T01:40:13.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:13.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:13.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_18 2022-04-10T01:40:13.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:40:13.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:40:13.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:13.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:13.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:13.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:13.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:40:13.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:40:14.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror image_18 started 2022-04-10T01:40:14.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:14.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:14.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_18 2022-04-10T01:40:14.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:40:14.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:40:14.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:40:14.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:14.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:14.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:14.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:14.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_18 2022-04-10T01:40:14.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:40:14.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:40:14.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:40:14.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:40:14.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:40:14.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:40:14.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:40:14.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:40:14.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:14.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:40:14.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:40:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:40:14.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:40:14.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:40:14.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:40:14.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_18 2022-04-10T01:40:14.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:40:14.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:40:14.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:40:14.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:40:14.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:40:14.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:40:14.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:40:14.556 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:40:14.556 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:40:14.556 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:40:14.556 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:40:14.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:40:14.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:40:14.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:40:14.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:40:14.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:40:14.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:14.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:40:14.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:40:14.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:40:14.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:40:14.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:40:14.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:40:14.562 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:40:14.562 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:40:14.562 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:40:14.562 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:40:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:40:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:40:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:40:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:40:14.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:40:14.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:40:14.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:40:14.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:40:14.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:14.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:40:14.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:40:14.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:40:14.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:40:14.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:40:14.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_18 2022-04-10T01:40:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:40:14.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:40:14.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:40:14.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:40:14.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:40:14.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:40:14.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:40:14.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:14.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_18 2022-04-10T01:40:14.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:40:14.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:14.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:14.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_18 2022-04-10T01:40:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:40:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:40:14.698 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:40:14.698 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_18 2022-04-10T01:40:14.698 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:40:14.698 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:40:14.698 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:14.699 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_18 2022-04-10T01:40:14.699 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:14.699 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:14.699 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:14.699 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:14.700 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:40:14.700 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:14.700 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:14.700 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:14.700 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-image_18.status 2022-04-10T01:40:14.701 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-image_18.status 2022-04-10T01:40:14.701 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:40:14.701 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_18.status 2022-04-10T01:40:14.701 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-image_18.status 2022-04-10T01:40:14.702 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-image_18.status 2022-04-10T01:40:14.702 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:40:14.703 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_18.status 2022-04-10T01:40:14.703 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror image_18 2022-04-10T01:40:14.704 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:14.704 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:14.704 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_18 2022-04-10T01:40:14.704 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:40:14.704 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:14.704 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:14.705 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:14.705 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_18 2022-04-10T01:40:15.556 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 170 2022-04-10T01:40:15.568 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_18 /tmp/tmp.rgwz74WspX/cluster2-mirror-image_18.status 2022-04-10T01:40:15.568 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:15.568 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:15.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_18 2022-04-10T01:40:15.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_18.status 2022-04-10T01:40:15.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:40:15.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:15.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:15.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:15.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_18 --format xml 2022-04-10T01:40:15.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:40:15.610 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:40:15.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=170 2022-04-10T01:40:15.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:40:15.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:40:15.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:40:15.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_18.status 2022-04-10T01:40:15.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:15.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:15.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_18 2022-04-10T01:40:15.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_18.status 2022-04-10T01:40:15.816 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:15.816 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:15.816 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:15.816 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:15.816 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2022-04-10T01:40:15.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:40:15.859 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:40:15.859 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:40:15.861 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:40:15.861 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:40:15.861 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:40:15.862 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:40:15.862 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 170 = '' 2022-04-10T01:40:15.863 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:40:15.863 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:40:16.263 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_18.status 2022-04-10T01:40:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_18 2022-04-10T01:40:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_18.status 2022-04-10T01:40:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2022-04-10T01:40:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:40:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:40:16.311 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:40:16.312 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:40:16.312 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:40:16.313 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:40:16.313 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:40:16.314 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 170 = '' 2022-04-10T01:40:16.314 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:40:16.314 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:40:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_18.status 2022-04-10T01:40:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_18 2022-04-10T01:40:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_18.status 2022-04-10T01:40:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2022-04-10T01:40:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:40:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:40:17.161 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=170 2022-04-10T01:40:17.161 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 170 = 170 2022-04-10T01:40:17.162 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:17.162 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror image_18 snap 2022-04-10T01:40:17.162 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:17.162 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:17.163 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_18 2022-04-10T01:40:17.163 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:40:17.163 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:40:17.163 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:17.163 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:17.164 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:17.164 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:17.164 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:40:17.164 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:40:18.162 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/image_18@snap 2022-04-10T01:40:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_18': 2022-04-10T01:40:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:40:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:40:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 2 2022-04-10T01:40:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 11332c7bbef9 2022-04-10T01:40:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.11332c7bbef9 2022-04-10T01:40:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:40:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:40:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:40:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:40:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:31:08 2022 2022-04-10T01:40:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:31:08 2022 2022-04-10T01:40:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:31:08 2022 2022-04-10T01:40:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:40:18.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:40:18.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:40:18.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 0481cc7e-2bc7-4c1c-903d-89babd0e91d4 2022-04-10T01:40:18.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:40:18.208 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:18.208 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror image_18 snap 2022-04-10T01:40:18.208 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:18.208 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_18 2022-04-10T01:40:18.209 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:40:18.209 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:40:18.209 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_18.export 2022-04-10T01:40:18.209 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_18.export 2022-04-10T01:40:18.209 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_18.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_18.export 2022-04-10T01:40:18.210 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export image_18@snap /tmp/tmp.rgwz74WspX/cluster2-mirror-image_18.export 2022-04-10T01:40:18.387 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:40:18.390 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export image_18@snap /tmp/tmp.rgwz74WspX/cluster1-mirror-image_18.export 2022-04-10T01:40:18.630 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:40:18.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-image_18.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_18.export 2022-04-10T01:40:18.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_18.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_18.export 2022-04-10T01:40:18.735 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:18.736 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:40:18.736 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_19 2022-04-10T01:40:18.736 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror image_19 snap 2022-04-10T01:40:18.736 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:18.737 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:18.737 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_19 2022-04-10T01:40:18.737 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:40:18.737 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create image_19@snap --debug-rbd=20 --debug-journaler=20 2022-04-10T01:40:20.506 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror image_19 2022-04-10T01:40:20.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:20.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:20.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_19 2022-04-10T01:40:20.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:20.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:20.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:20.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:20.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror image_19 started 2022-04-10T01:40:20.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:20.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:20.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_19 2022-04-10T01:40:20.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:40:20.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:40:20.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:20.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:20.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:20.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:40:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:40:21.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror image_19 started 2022-04-10T01:40:21.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:21.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_19 2022-04-10T01:40:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:40:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:40:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:40:21.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:21.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:21.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:21.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:21.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_19 2022-04-10T01:40:21.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:40:21.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:40:21.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:40:21.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:40:21.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:40:21.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:40:21.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:40:21.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:40:21.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:21.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:40:21.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:40:21.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:40:21.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:40:21.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:40:21.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:40:21.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_19 2022-04-10T01:40:21.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:40:21.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:40:21.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:40:21.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:40:21.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:40:21.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:40:21.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:40:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:40:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:40:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:40:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:40:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:40:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:40:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:40:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:40:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:40:21.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:21.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:40:21.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:40:21.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:40:21.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:40:21.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:40:21.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:40:21.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:40:21.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:40:21.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:40:21.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:40:21.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:40:21.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:40:21.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:40:21.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:40:21.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:40:21.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:40:21.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:40:21.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:40:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:40:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:40:21.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:40:21.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:40:21.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:40:21.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_19 2022-04-10T01:40:21.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:40:21.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:40:21.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:40:21.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:40:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:40:21.660 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:40:21.660 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:40:21.660 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:21.660 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_19 2022-04-10T01:40:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:40:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_19 2022-04-10T01:40:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:40:21.662 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:21.662 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:21.662 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:21.662 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:40:21.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:40:21.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_19 2022-04-10T01:40:21.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:40:21.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:40:21.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_19 2022-04-10T01:40:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:21.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:40:21.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:21.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:21.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:21.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-image_19.status 2022-04-10T01:40:21.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-image_19.status 2022-04-10T01:40:21.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:40:21.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_19.status 2022-04-10T01:40:21.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-image_19.status 2022-04-10T01:40:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-image_19.status 2022-04-10T01:40:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:40:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_19.status 2022-04-10T01:40:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror image_19 2022-04-10T01:40:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_19 2022-04-10T01:40:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:40:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_19 2022-04-10T01:40:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 173 2022-04-10T01:40:22.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_19 /tmp/tmp.rgwz74WspX/cluster2-mirror-image_19.status 2022-04-10T01:40:22.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:22.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:22.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_19 2022-04-10T01:40:22.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_19.status 2022-04-10T01:40:22.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:40:22.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:22.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:22.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:22.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_19 --format xml 2022-04-10T01:40:22.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:40:22.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:40:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=173 2022-04-10T01:40:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:40:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:40:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:40:22.828 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_19.status 2022-04-10T01:40:22.828 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:22.829 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:22.829 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_19 2022-04-10T01:40:22.829 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_19.status 2022-04-10T01:40:22.829 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:22.830 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:22.830 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:22.830 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:22.830 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2022-04-10T01:40:22.831 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:40:22.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:40:22.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:40:22.876 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:40:22.877 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:40:22.877 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:40:22.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:40:22.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 173 = '' 2022-04-10T01:40:22.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:40:22.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:40:23.279 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_19.status 2022-04-10T01:40:23.279 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:23.280 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:23.280 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_19 2022-04-10T01:40:23.280 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_19.status 2022-04-10T01:40:23.280 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:23.281 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:23.281 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:23.281 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:23.281 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2022-04-10T01:40:23.281 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:40:23.326 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:40:23.327 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:40:23.328 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:40:23.328 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:40:23.328 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:40:23.329 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:40:23.329 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 173 = '' 2022-04-10T01:40:23.330 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:40:23.330 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:40:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_19.status 2022-04-10T01:40:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_19 2022-04-10T01:40:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_19.status 2022-04-10T01:40:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:24.133 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:24.133 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:24.133 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:24.133 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2022-04-10T01:40:24.133 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:40:24.174 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:40:24.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=173 2022-04-10T01:40:24.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 173 = 173 2022-04-10T01:40:24.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:24.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror image_19 snap 2022-04-10T01:40:24.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:24.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:24.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_19 2022-04-10T01:40:24.207 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:40:24.207 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:40:24.207 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:24.207 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:24.207 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:24.208 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:24.208 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:40:24.208 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:40:25.177 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/image_19@snap 2022-04-10T01:40:25.218 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_19': 2022-04-10T01:40:25.218 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:40:25.218 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:40:25.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 2 2022-04-10T01:40:25.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 118db757f37a 2022-04-10T01:40:25.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.118db757f37a 2022-04-10T01:40:25.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:40:25.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:40:25.220 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:40:25.220 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:40:25.220 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:31:18 2022 2022-04-10T01:40:25.220 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:31:18 2022 2022-04-10T01:40:25.221 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:31:18 2022 2022-04-10T01:40:25.221 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:40:25.221 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:40:25.221 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:40:25.221 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 45a317c4-845d-40d4-b7de-aa8c3ca7ce90 2022-04-10T01:40:25.222 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:40:25.223 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:25.223 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror image_19 snap 2022-04-10T01:40:25.224 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:25.224 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_19 2022-04-10T01:40:25.224 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:40:25.224 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:40:25.225 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_19.export 2022-04-10T01:40:25.225 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_19.export 2022-04-10T01:40:25.225 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_19.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_19.export 2022-04-10T01:40:25.225 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export image_19@snap /tmp/tmp.rgwz74WspX/cluster2-mirror-image_19.export 2022-04-10T01:40:25.408 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:40:25.411 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export image_19@snap /tmp/tmp.rgwz74WspX/cluster1-mirror-image_19.export 2022-04-10T01:40:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:40:25.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-image_19.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_19.export 2022-04-10T01:40:25.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_19.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_19.export 2022-04-10T01:40:25.747 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:25.747 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:40:25.748 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_20 2022-04-10T01:40:25.748 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror image_20 snap 2022-04-10T01:40:25.748 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:25.748 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:25.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_20 2022-04-10T01:40:25.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:40:25.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create image_20@snap --debug-rbd=20 --debug-journaler=20 2022-04-10T01:40:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror image_20 2022-04-10T01:40:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_20 2022-04-10T01:40:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror image_20 started 2022-04-10T01:40:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_20 2022-04-10T01:40:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:40:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:40:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:27.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:27.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:40:27.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:40:28.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror image_20 started 2022-04-10T01:40:28.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:28.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:28.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_20 2022-04-10T01:40:28.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:40:28.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:40:28.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:40:28.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:28.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:28.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:28.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:28.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_20 2022-04-10T01:40:28.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:40:28.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:40:28.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:40:28.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:40:28.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:40:28.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:40:28.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:40:28.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:40:28.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:28.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:40:28.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:40:28.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:40:28.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:40:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:40:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:40:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2022-04-10T01:40:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:40:28.587 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:40:28.587 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:40:28.587 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:40:28.587 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:40:28.587 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:40:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:40:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:40:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:40:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:40:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:40:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:40:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:40:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:40:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:40:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:40:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:40:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:40:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:40:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:40:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:40:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:40:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:40:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:40:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:40:28.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:40:28.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:40:28.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:40:28.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:40:28.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:40:28.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:40:28.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:40:28.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:40:28.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:40:28.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:28.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:40:28.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:40:28.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:40:28.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:40:28.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:40:28.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2022-04-10T01:40:28.712 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2022-04-10T01:40:28.712 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:40:28.712 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:40:28.716 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:40:28.716 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_20 2022-04-10T01:40:28.716 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:1 2022-04-10T01:40:28.717 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:40:28.717 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:40:28.717 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:1 cluster instance 2022-04-10T01:40:28.717 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:1 2022-04-10T01:40:28.717 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:40:28.718 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:40:28.718 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:40:28.718 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=1 2022-04-10T01:40:28.718 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 1 = cluster1:1 ']' 2022-04-10T01:40:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:40:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:40:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=1 2022-04-10T01:40:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:40:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:40:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:40:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:40:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:1 cluster1 2022-04-10T01:40:28.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:1 2022-04-10T01:40:28.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:40:28.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:40:28.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2022-04-10T01:40:28.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:1 2022-04-10T01:40:28.722 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:40:28.722 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:40:28.722 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:40:28.722 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:40:28.722 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 1 = cluster1:1 ']' 2022-04-10T01:40:28.723 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:40:28.723 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:40:28.723 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=1 2022-04-10T01:40:28.723 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=1 2022-04-10T01:40:28.724 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:40:28.724 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:40:28.724 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:28.724 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:40:28.724 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2022-04-10T01:40:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:40:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:40:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:40:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_20 2022-04-10T01:40:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2022-04-10T01:40:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:40:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:40:28.833 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:40:28.833 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_20 2022-04-10T01:40:28.833 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:2 2022-04-10T01:40:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:40:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:40:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:2 cluster instance 2022-04-10T01:40:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:2 2022-04-10T01:40:28.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:40:28.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:40:28.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:40:28.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=2 2022-04-10T01:40:28.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 2 = cluster1:2 ']' 2022-04-10T01:40:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:40:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:40:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=2 2022-04-10T01:40:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:40:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:40:28.837 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:40:28.837 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:28.837 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:40:28.837 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:2 cluster1 2022-04-10T01:40:28.837 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:2 2022-04-10T01:40:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:40:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:40:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2022-04-10T01:40:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:2 2022-04-10T01:40:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:40:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:40:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:40:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:40:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 2 = cluster1:2 ']' 2022-04-10T01:40:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:40:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:40:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=2 2022-04-10T01:40:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=2 2022-04-10T01:40:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:40:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:40:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:40:28.842 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2022-04-10T01:40:28.847 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:40:28.847 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:40:28.847 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:40:28.848 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_20 2022-04-10T01:40:28.951 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2022-04-10T01:40:28.951 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:40:28.951 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:40:28.955 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:40:28.955 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_20 2022-04-10T01:40:28.955 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:3 2022-04-10T01:40:28.956 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:40:28.956 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:40:28.956 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:3 cluster instance 2022-04-10T01:40:28.956 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:3 2022-04-10T01:40:28.956 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:40:28.956 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:40:28.957 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:40:28.957 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=3 2022-04-10T01:40:28.957 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 3 = cluster1:3 ']' 2022-04-10T01:40:28.957 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:40:28.957 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:40:28.958 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=3 2022-04-10T01:40:28.958 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=3 2022-04-10T01:40:28.958 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:40:28.958 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:40:28.959 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:28.959 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:40:28.959 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:3 cluster1 2022-04-10T01:40:28.959 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:3 2022-04-10T01:40:28.959 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:40:28.960 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:40:28.960 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2022-04-10T01:40:28.960 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:3 2022-04-10T01:40:28.960 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:40:28.960 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:40:28.961 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:40:28.961 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=3 2022-04-10T01:40:28.961 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 3 = cluster1:3 ']' 2022-04-10T01:40:28.961 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:40:28.961 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:40:28.962 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=3 2022-04-10T01:40:28.962 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=3 2022-04-10T01:40:28.962 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:40:28.962 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:40:28.963 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:28.963 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:40:28.963 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2022-04-10T01:40:28.969 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:40:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:40:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:40:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_20 2022-04-10T01:40:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:40:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:40:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:40:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:40:29.084 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:40:29.088 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:40:29.088 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:40:29.088 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:29.088 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_20 2022-04-10T01:40:29.089 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:40:29.089 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:29.089 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:29.089 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_20 2022-04-10T01:40:29.089 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:40:29.090 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:29.090 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:29.090 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:29.090 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:40:29.090 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:40:29.091 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_20 2022-04-10T01:40:29.091 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:40:29.091 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:40:29.091 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:29.092 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_20 2022-04-10T01:40:29.092 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:29.092 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:29.092 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:29.092 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:29.093 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:40:29.093 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:29.093 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:29.093 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:29.093 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-image_20.status 2022-04-10T01:40:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-image_20.status 2022-04-10T01:40:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:40:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_20.status 2022-04-10T01:40:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-image_20.status 2022-04-10T01:40:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-image_20.status 2022-04-10T01:40:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:40:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_20.status 2022-04-10T01:40:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror image_20 2022-04-10T01:40:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_20 2022-04-10T01:40:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:40:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:29.097 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:29.097 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_20 2022-04-10T01:40:29.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 176 2022-04-10T01:40:29.585 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_20 /tmp/tmp.rgwz74WspX/cluster2-mirror-image_20.status 2022-04-10T01:40:29.585 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:29.586 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:29.586 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_20 2022-04-10T01:40:29.586 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_20.status 2022-04-10T01:40:29.586 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:40:29.587 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:29.587 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:29.587 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:29.587 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_20 --format xml 2022-04-10T01:40:29.587 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:40:29.627 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:40:29.629 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=176 2022-04-10T01:40:29.629 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:40:29.629 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:40:29.630 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:40:29.831 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_20.status 2022-04-10T01:40:29.831 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:29.831 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:29.831 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_20 2022-04-10T01:40:29.832 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_20.status 2022-04-10T01:40:29.832 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:29.832 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:29.832 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:29.832 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:29.833 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2022-04-10T01:40:29.833 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:40:29.874 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:40:29.874 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:40:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:40:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:40:29.876 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:40:29.876 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:40:29.877 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 176 = '' 2022-04-10T01:40:29.877 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:40:29.877 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:40:30.278 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_20.status 2022-04-10T01:40:30.278 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:30.278 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:30.278 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_20 2022-04-10T01:40:30.279 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_20.status 2022-04-10T01:40:30.279 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:30.279 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:30.279 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:30.279 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:30.280 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2022-04-10T01:40:30.280 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:40:30.325 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:40:30.326 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:40:30.327 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:40:30.328 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:40:30.328 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:40:30.329 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:40:30.329 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 176 = '' 2022-04-10T01:40:30.329 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:40:30.329 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:40:30.534 INFO:tasks.ceph.cluster2.osd.2.smithi043.stderr:2022-04-10T01:40:30.531+0000 7f8be0453700 -1 reset not still connected to 0x55f2a5aa8f70 2022-04-10T01:40:30.535 INFO:tasks.ceph.cluster2.osd.2.smithi043.stderr:2022-04-10T01:40:30.531+0000 7f8be0453700 -1 reset not still connected to 0x55f2b4f631e0 2022-04-10T01:40:30.535 INFO:tasks.ceph.cluster2.osd.2.smithi043.stderr:2022-04-10T01:40:30.531+0000 7f8be0453700 -1 reset not still connected to 0x55f2b95620d0 2022-04-10T01:40:30.535 INFO:tasks.ceph.cluster2.osd.2.smithi043.stderr:2022-04-10T01:40:30.531+0000 7f8be0453700 -1 reset not still connected to 0x55f2beff92b0 2022-04-10T01:40:31.130 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_20.status 2022-04-10T01:40:31.130 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:31.131 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:31.131 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_20 2022-04-10T01:40:31.131 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_20.status 2022-04-10T01:40:31.131 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2022-04-10T01:40:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:40:31.171 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:40:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=176 2022-04-10T01:40:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 176 = 176 2022-04-10T01:40:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror image_20 snap 2022-04-10T01:40:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_20 2022-04-10T01:40:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:40:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:40:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:31.177 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:31.177 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:40:31.177 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:40:32.174 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/image_20@snap 2022-04-10T01:40:32.216 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_20': 2022-04-10T01:40:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:40:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:40:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 2 2022-04-10T01:40:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 12a74041e3a9 2022-04-10T01:40:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.12a74041e3a9 2022-04-10T01:40:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:40:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:40:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:40:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:40:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:31:27 2022 2022-04-10T01:40:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:31:27 2022 2022-04-10T01:40:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:31:27 2022 2022-04-10T01:40:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:40:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:40:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:40:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 828304b8-d69a-488c-b09a-e6569fe239bd 2022-04-10T01:40:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:40:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror image_20 snap 2022-04-10T01:40:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_20 2022-04-10T01:40:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:40:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:40:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_20.export 2022-04-10T01:40:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_20.export 2022-04-10T01:40:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_20.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_20.export 2022-04-10T01:40:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export image_20@snap /tmp/tmp.rgwz74WspX/cluster2-mirror-image_20.export 2022-04-10T01:40:32.409 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:40:32.412 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export image_20@snap /tmp/tmp.rgwz74WspX/cluster1-mirror-image_20.export 2022-04-10T01:40:32.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:40:32.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-image_20.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_20.export 2022-04-10T01:40:32.746 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_20.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_20.export 2022-04-10T01:40:32.784 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:32.784 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:40:32.784 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_21 2022-04-10T01:40:32.784 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror image_21 snap 2022-04-10T01:40:32.785 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:32.785 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:32.785 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_21 2022-04-10T01:40:32.785 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:40:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create image_21@snap --debug-rbd=20 --debug-journaler=20 2022-04-10T01:40:34.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror image_21 2022-04-10T01:40:34.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:34.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:34.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_21 2022-04-10T01:40:34.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:34.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:34.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror image_21 started 2022-04-10T01:40:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_21 2022-04-10T01:40:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:40:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:40:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:40:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:40:35.506 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror image_21 started 2022-04-10T01:40:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_21 2022-04-10T01:40:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:40:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:40:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:40:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_21 2022-04-10T01:40:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:40:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:40:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:40:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:40:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:40:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:40:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:40:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:40:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:40:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:40:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:40:35.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:40:35.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:40:35.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:40:35.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_21 2022-04-10T01:40:35.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:40:35.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:40:35.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:40:35.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:40:35.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:40:35.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:40:35.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:40:35.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:40:35.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:40:35.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:40:35.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:40:35.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:40:35.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:40:35.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:40:35.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:40:35.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:40:35.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:35.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:40:35.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:40:35.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:40:35.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:40:35.528 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:40:35.528 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:40:35.529 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:40:35.529 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:40:35.530 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:40:35.530 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:40:35.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:40:35.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:40:35.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:40:35.532 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:40:35.532 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:40:35.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:40:35.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:40:35.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:40:35.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:35.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:40:35.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:40:35.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:40:35.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:40:35.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:40:35.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_21 2022-04-10T01:40:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:40:35.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:40:35.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:40:35.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:40:35.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:40:35.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:40:35.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:40:35.658 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:35.658 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_21 2022-04-10T01:40:35.658 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:40:35.658 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:35.659 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:35.659 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_21 2022-04-10T01:40:35.659 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:40:35.659 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:35.660 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:35.660 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:35.660 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:40:35.660 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:40:35.660 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_21 2022-04-10T01:40:35.661 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:40:35.661 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:40:35.661 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:35.661 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_21 2022-04-10T01:40:35.662 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:35.662 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:35.662 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:35.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:35.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:40:35.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:35.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:35.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:35.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-image_21.status 2022-04-10T01:40:35.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-image_21.status 2022-04-10T01:40:35.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:40:35.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_21.status 2022-04-10T01:40:35.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-image_21.status 2022-04-10T01:40:35.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-image_21.status 2022-04-10T01:40:35.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:40:35.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_21.status 2022-04-10T01:40:35.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror image_21 2022-04-10T01:40:35.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:35.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:35.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_21 2022-04-10T01:40:35.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:40:35.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:35.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:35.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:35.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_21 2022-04-10T01:40:36.584 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 179 2022-04-10T01:40:36.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_21 /tmp/tmp.rgwz74WspX/cluster2-mirror-image_21.status 2022-04-10T01:40:36.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:36.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:36.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_21 2022-04-10T01:40:36.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_21.status 2022-04-10T01:40:36.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:40:36.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:36.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:36.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:36.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_21 --format xml 2022-04-10T01:40:36.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:40:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:40:36.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=179 2022-04-10T01:40:36.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:40:36.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:40:36.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:40:36.842 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_21.status 2022-04-10T01:40:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_21 2022-04-10T01:40:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_21.status 2022-04-10T01:40:36.844 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:36.844 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:36.844 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:36.844 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:36.844 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2022-04-10T01:40:36.845 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:40:36.889 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:40:36.889 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:40:36.890 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:40:36.891 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:40:36.891 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:40:36.892 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:40:36.892 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 179 = '' 2022-04-10T01:40:36.892 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:40:36.892 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:40:37.293 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_21.status 2022-04-10T01:40:37.293 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:37.294 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:37.294 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_21 2022-04-10T01:40:37.294 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_21.status 2022-04-10T01:40:37.294 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:37.295 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:37.295 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:37.295 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:37.295 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2022-04-10T01:40:37.296 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:40:37.341 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:40:37.342 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:40:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:40:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:40:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:40:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:40:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 179 = '' 2022-04-10T01:40:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:40:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:40:38.145 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_21.status 2022-04-10T01:40:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_21 2022-04-10T01:40:38.147 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_21.status 2022-04-10T01:40:38.147 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:38.147 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:38.147 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:38.147 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:38.148 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2022-04-10T01:40:38.148 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:40:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:40:38.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=179 2022-04-10T01:40:38.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 179 = 179 2022-04-10T01:40:38.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:38.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror image_21 snap 2022-04-10T01:40:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_21 2022-04-10T01:40:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:40:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:40:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:40:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:40:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/image_21@snap 2022-04-10T01:40:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_21': 2022-04-10T01:40:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:40:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:40:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 2 2022-04-10T01:40:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 10e46537d8e0 2022-04-10T01:40:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.10e46537d8e0 2022-04-10T01:40:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:40:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:40:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:40:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:40:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:31:36 2022 2022-04-10T01:40:39.233 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:31:36 2022 2022-04-10T01:40:39.233 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:31:36 2022 2022-04-10T01:40:39.233 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:40:39.233 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:40:39.234 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:40:39.234 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 65c94954-3314-4812-9697-85faa0cd8459 2022-04-10T01:40:39.234 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:40:39.235 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:39.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror image_21 snap 2022-04-10T01:40:39.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:39.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_21 2022-04-10T01:40:39.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:40:39.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:40:39.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_21.export 2022-04-10T01:40:39.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_21.export 2022-04-10T01:40:39.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_21.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_21.export 2022-04-10T01:40:39.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export image_21@snap /tmp/tmp.rgwz74WspX/cluster2-mirror-image_21.export 2022-04-10T01:40:39.414 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:40:39.417 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export image_21@snap /tmp/tmp.rgwz74WspX/cluster1-mirror-image_21.export 2022-04-10T01:40:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:40:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-image_21.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_21.export 2022-04-10T01:40:39.737 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_21.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_21.export 2022-04-10T01:40:39.775 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:39.775 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:40:39.775 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_22 2022-04-10T01:40:39.775 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror image_22 snap 2022-04-10T01:40:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_22 2022-04-10T01:40:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:40:39.777 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create image_22@snap --debug-rbd=20 --debug-journaler=20 2022-04-10T01:40:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror image_22 2022-04-10T01:40:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_22 2022-04-10T01:40:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror image_22 started 2022-04-10T01:40:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_22 2022-04-10T01:40:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:40:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:40:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:40:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:40:42.605 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror image_22 started 2022-04-10T01:40:42.606 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:42.606 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:42.606 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_22 2022-04-10T01:40:42.607 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:40:42.607 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:40:42.607 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:40:42.607 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:42.607 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:42.608 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:42.608 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:42.608 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_22 2022-04-10T01:40:42.608 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:40:42.608 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:40:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:40:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:40:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:40:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:40:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:40:42.610 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:40:42.610 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:42.610 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:40:42.610 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:40:42.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:40:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:40:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:40:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:40:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_22 2022-04-10T01:40:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:40:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:40:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:40:42.616 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:40:42.616 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:40:42.616 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:40:42.616 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:40:42.617 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:40:42.617 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:40:42.617 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:40:42.617 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:40:42.617 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:40:42.618 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:40:42.618 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:40:42.618 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:40:42.618 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:40:42.618 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:42.619 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:40:42.619 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:40:42.619 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:40:42.619 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:40:42.619 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:40:42.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:40:42.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:40:42.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:40:42.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:40:42.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:40:42.621 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:40:42.621 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:40:42.621 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:40:42.621 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:40:42.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:40:42.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:40:42.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:40:42.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:40:42.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:42.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:40:42.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:40:42.627 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:40:42.627 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:40:42.627 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:40:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_22 2022-04-10T01:40:42.740 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2022-04-10T01:40:42.740 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:40:42.741 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:40:42.744 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:40:42.744 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_22 2022-04-10T01:40:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:1 2022-04-10T01:40:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:40:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:40:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:1 cluster instance 2022-04-10T01:40:42.746 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:1 2022-04-10T01:40:42.746 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:40:42.746 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:40:42.746 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:40:42.746 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=1 2022-04-10T01:40:42.747 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 1 = cluster1:1 ']' 2022-04-10T01:40:42.747 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:40:42.747 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:40:42.747 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=1 2022-04-10T01:40:42.747 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:40:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:40:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:40:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:40:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:1 cluster1 2022-04-10T01:40:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:1 2022-04-10T01:40:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:40:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:40:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2022-04-10T01:40:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:1 2022-04-10T01:40:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:40:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:40:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:40:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:40:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 1 = cluster1:1 ']' 2022-04-10T01:40:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:40:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:40:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=1 2022-04-10T01:40:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=1 2022-04-10T01:40:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:40:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:40:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:40:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2022-04-10T01:40:42.759 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:40:42.759 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:40:42.759 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:40:42.760 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_22 2022-04-10T01:40:42.864 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2022-04-10T01:40:42.864 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:40:42.864 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:40:42.868 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:40:42.868 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_22 2022-04-10T01:40:42.869 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:2 2022-04-10T01:40:42.869 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:40:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:40:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:2 cluster instance 2022-04-10T01:40:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:2 2022-04-10T01:40:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:40:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:40:42.872 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:40:42.872 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=2 2022-04-10T01:40:42.872 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 2 = cluster1:2 ']' 2022-04-10T01:40:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:40:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:40:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=2 2022-04-10T01:40:42.874 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:40:42.874 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:40:42.874 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:40:42.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:42.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:40:42.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:2 cluster1 2022-04-10T01:40:42.876 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:2 2022-04-10T01:40:42.876 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:40:42.876 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:40:42.876 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2022-04-10T01:40:42.877 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:2 2022-04-10T01:40:42.877 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:40:42.877 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:40:42.877 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:40:42.877 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:40:42.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 2 = cluster1:2 ']' 2022-04-10T01:40:42.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:40:42.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:40:42.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=2 2022-04-10T01:40:42.879 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=2 2022-04-10T01:40:42.879 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:40:42.879 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:40:42.879 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:42.879 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:40:42.880 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2022-04-10T01:40:42.882 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:40:42.882 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:40:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:40:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_22 2022-04-10T01:40:42.988 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2022-04-10T01:40:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:40:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:40:42.992 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:40:42.992 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_22 2022-04-10T01:40:42.993 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:3 2022-04-10T01:40:42.993 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:40:42.993 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:40:42.993 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:3 cluster instance 2022-04-10T01:40:42.993 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:3 2022-04-10T01:40:42.994 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:40:42.994 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:40:42.994 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:40:42.994 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=3 2022-04-10T01:40:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 3 = cluster1:3 ']' 2022-04-10T01:40:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:40:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:40:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=3 2022-04-10T01:40:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=3 2022-04-10T01:40:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:40:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:40:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:40:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:3 cluster1 2022-04-10T01:40:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:3 2022-04-10T01:40:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:40:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:40:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2022-04-10T01:40:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:3 2022-04-10T01:40:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:40:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:40:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:40:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=3 2022-04-10T01:40:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 3 = cluster1:3 ']' 2022-04-10T01:40:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:40:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:40:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=3 2022-04-10T01:40:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=3 2022-04-10T01:40:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:40:43.000 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:40:43.000 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:43.000 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:40:43.000 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2022-04-10T01:40:43.008 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:40:43.008 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:40:43.008 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:40:43.009 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_22 2022-04-10T01:40:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:40:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:40:43.126 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:40:43.126 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:40:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:40:43.130 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:40:43.130 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:40:43.130 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:43.130 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_22 2022-04-10T01:40:43.130 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:40:43.131 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:43.131 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:43.131 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_22 2022-04-10T01:40:43.131 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:40:43.131 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:43.132 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:43.132 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:43.132 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:40:43.132 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:40:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_22 2022-04-10T01:40:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:40:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:40:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_22 2022-04-10T01:40:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:43.134 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:43.134 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:43.134 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:43.134 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:40:43.135 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:43.135 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:43.135 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:43.135 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-image_22.status 2022-04-10T01:40:43.135 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-image_22.status 2022-04-10T01:40:43.136 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:40:43.136 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_22.status 2022-04-10T01:40:43.136 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-image_22.status 2022-04-10T01:40:43.136 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-image_22.status 2022-04-10T01:40:43.136 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:40:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_22.status 2022-04-10T01:40:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror image_22 2022-04-10T01:40:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_22 2022-04-10T01:40:43.139 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:40:43.139 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:43.139 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:43.139 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:43.139 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_22 2022-04-10T01:40:43.603 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 182 2022-04-10T01:40:43.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_22 /tmp/tmp.rgwz74WspX/cluster2-mirror-image_22.status 2022-04-10T01:40:43.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:43.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:43.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_22 2022-04-10T01:40:43.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_22.status 2022-04-10T01:40:43.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:40:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_22 --format xml 2022-04-10T01:40:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:40:43.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:40:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=182 2022-04-10T01:40:43.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:40:43.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:40:43.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:40:43.856 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_22.status 2022-04-10T01:40:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_22 2022-04-10T01:40:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_22.status 2022-04-10T01:40:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:43.858 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:43.858 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:43.858 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:43.858 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2022-04-10T01:40:43.859 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:40:43.900 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:40:43.901 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:40:43.902 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:40:43.902 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:40:43.902 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:40:43.903 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:40:43.903 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 182 = '' 2022-04-10T01:40:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:40:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:40:44.306 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_22.status 2022-04-10T01:40:44.306 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:44.307 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:44.307 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_22 2022-04-10T01:40:44.308 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_22.status 2022-04-10T01:40:44.308 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:44.308 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2022-04-10T01:40:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:40:44.354 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:40:44.355 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:40:44.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:40:44.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:40:44.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:40:44.357 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:40:44.357 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 182 = '' 2022-04-10T01:40:44.357 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:40:44.358 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:40:45.158 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_22.status 2022-04-10T01:40:45.159 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:45.159 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:45.159 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_22 2022-04-10T01:40:45.160 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_22.status 2022-04-10T01:40:45.160 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:45.160 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:45.160 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:45.160 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:45.160 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2022-04-10T01:40:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:40:45.201 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:40:45.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=182 2022-04-10T01:40:45.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 182 = 182 2022-04-10T01:40:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror image_22 snap 2022-04-10T01:40:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_22 2022-04-10T01:40:45.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:40:45.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:40:45.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:45.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:45.207 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:45.207 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:45.207 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:40:45.207 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:40:46.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/image_22@snap 2022-04-10T01:40:46.247 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_22': 2022-04-10T01:40:46.247 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:40:46.248 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:40:46.248 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 2 2022-04-10T01:40:46.248 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 113370f8b024 2022-04-10T01:40:46.248 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.113370f8b024 2022-04-10T01:40:46.248 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:40:46.249 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:40:46.249 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:40:46.249 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:40:46.249 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:31:42 2022 2022-04-10T01:40:46.249 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:31:42 2022 2022-04-10T01:40:46.250 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:31:42 2022 2022-04-10T01:40:46.250 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:40:46.250 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:40:46.250 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:40:46.250 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: be4c9569-5fe3-47a0-a0a1-84d380d56af1 2022-04-10T01:40:46.251 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:40:46.252 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:46.252 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror image_22 snap 2022-04-10T01:40:46.252 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:46.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_22 2022-04-10T01:40:46.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:40:46.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:40:46.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_22.export 2022-04-10T01:40:46.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_22.export 2022-04-10T01:40:46.254 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_22.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_22.export 2022-04-10T01:40:46.254 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export image_22@snap /tmp/tmp.rgwz74WspX/cluster2-mirror-image_22.export 2022-04-10T01:40:46.435 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:40:46.439 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export image_22@snap /tmp/tmp.rgwz74WspX/cluster1-mirror-image_22.export 2022-04-10T01:40:46.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:40:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-image_22.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_22.export 2022-04-10T01:40:46.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_22.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_22.export 2022-04-10T01:40:46.786 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:46.787 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:40:46.787 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_23 2022-04-10T01:40:46.787 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror image_23 snap 2022-04-10T01:40:46.787 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_23 2022-04-10T01:40:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:40:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create image_23@snap --debug-rbd=20 --debug-journaler=20 2022-04-10T01:40:48.609 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror image_23 2022-04-10T01:40:48.609 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:48.610 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:48.610 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_23 2022-04-10T01:40:48.610 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:48.610 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror image_23 started 2022-04-10T01:40:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_23 2022-04-10T01:40:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:40:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:40:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:48.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:48.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:40:48.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:40:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror image_23 started 2022-04-10T01:40:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:49.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_23 2022-04-10T01:40:49.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:40:49.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:40:49.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:40:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_23 2022-04-10T01:40:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:40:49.614 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:40:49.614 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:40:49.614 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:40:49.614 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:40:49.615 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:40:49.615 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:40:49.615 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:40:49.615 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:49.615 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:40:49.616 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:40:49.616 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:40:49.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:40:49.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:40:49.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:40:49.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_23 2022-04-10T01:40:49.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:40:49.621 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:40:49.621 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:40:49.621 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:40:49.621 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:40:49.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:40:49.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:40:49.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:40:49.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:40:49.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:40:49.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:40:49.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:40:49.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:40:49.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:40:49.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:40:49.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:40:49.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:49.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:40:49.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:40:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:40:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:40:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:40:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:40:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:40:49.626 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:40:49.626 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:40:49.626 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:40:49.626 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:40:49.627 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:40:49.627 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:40:49.627 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:40:49.627 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:40:49.627 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:40:49.628 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:40:49.628 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:40:49.628 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:49.628 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:40:49.628 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:40:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:40:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:40:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:40:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_23 2022-04-10T01:40:49.748 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2022-04-10T01:40:49.824 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:40:49.824 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:40:49.828 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:40:49.829 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_23 2022-04-10T01:40:49.829 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:1 2022-04-10T01:40:49.829 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:40:49.829 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:40:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:1 cluster instance 2022-04-10T01:40:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:1 2022-04-10T01:40:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:40:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:40:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:40:49.831 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=1 2022-04-10T01:40:49.831 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 1 = cluster1:1 ']' 2022-04-10T01:40:49.831 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:40:49.831 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:40:49.832 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=1 2022-04-10T01:40:49.832 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:40:49.832 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:40:49.832 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:40:49.832 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:49.833 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:40:49.833 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:1 cluster1 2022-04-10T01:40:49.833 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:1 2022-04-10T01:40:49.833 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:40:49.833 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:40:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2022-04-10T01:40:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:1 2022-04-10T01:40:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:40:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:40:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:40:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:40:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 1 = cluster1:1 ']' 2022-04-10T01:40:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:40:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:40:49.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=1 2022-04-10T01:40:49.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=1 2022-04-10T01:40:49.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:40:49.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:40:49.837 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:49.837 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:40:49.837 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2022-04-10T01:40:49.837 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:40:49.837 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:40:49.838 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:40:49.838 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_23 2022-04-10T01:40:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:40:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:40:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:40:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:40:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:40:49.885 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:40:49.886 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:40:49.886 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:49.886 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_23 2022-04-10T01:40:49.886 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:40:49.887 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:49.887 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:49.887 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_23 2022-04-10T01:40:49.887 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:40:49.888 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:49.888 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:49.888 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:49.888 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:40:49.888 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:40:49.889 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_23 2022-04-10T01:40:49.889 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:40:49.889 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:40:49.889 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:49.889 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_23 2022-04-10T01:40:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:40:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-image_23.status 2022-04-10T01:40:49.892 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-image_23.status 2022-04-10T01:40:49.892 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:40:49.892 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_23.status 2022-04-10T01:40:49.892 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-image_23.status 2022-04-10T01:40:49.892 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-image_23.status 2022-04-10T01:40:49.893 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:40:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_23.status 2022-04-10T01:40:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror image_23 2022-04-10T01:40:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:49.895 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:49.895 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_23 2022-04-10T01:40:49.895 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:40:49.895 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:49.896 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:49.896 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:49.896 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_23 2022-04-10T01:40:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 185 2022-04-10T01:40:50.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_23 /tmp/tmp.rgwz74WspX/cluster2-mirror-image_23.status 2022-04-10T01:40:50.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_23 2022-04-10T01:40:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_23.status 2022-04-10T01:40:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:40:50.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:50.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:50.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:50.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_23 --format xml 2022-04-10T01:40:50.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:40:50.718 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:40:50.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=185 2022-04-10T01:40:50.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:40:50.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:40:50.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:40:50.922 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_23.status 2022-04-10T01:40:50.923 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:50.923 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:50.923 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_23 2022-04-10T01:40:50.923 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_23.status 2022-04-10T01:40:50.924 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:50.924 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:50.924 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:50.924 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:50.924 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2022-04-10T01:40:50.925 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:40:50.975 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:40:50.975 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:40:50.976 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:40:50.976 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:40:50.977 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:40:50.977 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:40:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 185 = '' 2022-04-10T01:40:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:40:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:40:51.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_23.status 2022-04-10T01:40:51.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:51.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_23 2022-04-10T01:40:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_23.status 2022-04-10T01:40:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2022-04-10T01:40:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:40:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:40:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:40:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:40:51.437 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:40:51.437 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:40:51.438 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:40:51.438 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 185 = '' 2022-04-10T01:40:51.439 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:40:51.439 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:40:52.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_23.status 2022-04-10T01:40:52.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:52.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:52.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_23 2022-04-10T01:40:52.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_23.status 2022-04-10T01:40:52.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:52.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:52.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:52.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:52.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2022-04-10T01:40:52.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:40:52.281 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:40:52.283 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=185 2022-04-10T01:40:52.284 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 185 = 185 2022-04-10T01:40:52.284 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:52.284 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror image_23 snap 2022-04-10T01:40:52.284 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_23 2022-04-10T01:40:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:40:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:40:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:40:52.287 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:40:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/image_23@snap 2022-04-10T01:40:53.327 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_23': 2022-04-10T01:40:53.327 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:40:53.327 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:40:53.328 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 2 2022-04-10T01:40:53.328 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 118da1601ea 2022-04-10T01:40:53.328 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.118da1601ea 2022-04-10T01:40:53.328 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:40:53.328 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:40:53.329 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:40:53.329 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:40:53.329 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:31:56 2022 2022-04-10T01:40:53.329 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:31:56 2022 2022-04-10T01:40:53.330 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:31:56 2022 2022-04-10T01:40:53.330 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:40:53.330 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:40:53.330 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:40:53.330 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 656c53f5-50a2-44ea-8dbc-608ddbbbe7f5 2022-04-10T01:40:53.331 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:40:53.332 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:53.332 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror image_23 snap 2022-04-10T01:40:53.332 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:53.333 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_23 2022-04-10T01:40:53.333 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:40:53.333 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:40:53.333 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_23.export 2022-04-10T01:40:53.334 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_23.export 2022-04-10T01:40:53.334 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_23.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_23.export 2022-04-10T01:40:53.334 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export image_23@snap /tmp/tmp.rgwz74WspX/cluster2-mirror-image_23.export 2022-04-10T01:40:53.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:40:53.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export image_23@snap /tmp/tmp.rgwz74WspX/cluster1-mirror-image_23.export 2022-04-10T01:40:53.726 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:40:53.729 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-image_23.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_23.export 2022-04-10T01:40:53.793 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_23.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_23.export 2022-04-10T01:40:53.830 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:53.830 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:40:53.831 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_24 2022-04-10T01:40:53.831 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror image_24 snap 2022-04-10T01:40:53.831 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:53.831 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:53.831 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_24 2022-04-10T01:40:53.832 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:40:53.832 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create image_24@snap --debug-rbd=20 --debug-journaler=20 2022-04-10T01:40:55.506 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror image_24 2022-04-10T01:40:55.506 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:55.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:55.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_24 2022-04-10T01:40:55.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:55.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:55.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:55.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:55.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror image_24 started 2022-04-10T01:40:55.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:55.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:55.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_24 2022-04-10T01:40:55.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:40:55.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:40:55.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:55.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:55.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:55.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:55.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:40:55.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:40:56.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror image_24 started 2022-04-10T01:40:56.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:56.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:56.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_24 2022-04-10T01:40:56.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:40:56.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:40:56.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:40:56.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:56.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:56.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:56.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:56.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_24 2022-04-10T01:40:56.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:40:56.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:40:56.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:40:56.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:40:56.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:40:56.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:40:56.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:40:56.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:40:56.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:56.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:40:56.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:40:56.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:40:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:40:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:40:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:40:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2022-04-10T01:40:56.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:40:56.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:40:56.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:40:56.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:40:56.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:40:56.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:40:56.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:40:56.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:40:56.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:40:56.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:40:56.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:40:56.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:40:56.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:40:56.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:40:56.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:40:56.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:40:56.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:56.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:40:56.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:40:56.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:40:56.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:40:56.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:40:56.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:40:56.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:40:56.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:40:56.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:40:56.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:40:56.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:40:56.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:40:56.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:40:56.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:40:56.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:40:56.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:40:56.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:40:56.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:40:56.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:56.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:40:56.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:40:56.530 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:40:56.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:40:56.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:40:56.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2022-04-10T01:40:56.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2022-04-10T01:40:56.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:40:56.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:40:56.649 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:40:56.649 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_24 2022-04-10T01:40:56.649 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:1 2022-04-10T01:40:56.650 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:40:56.650 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:40:56.650 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:1 cluster instance 2022-04-10T01:40:56.650 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:1 2022-04-10T01:40:56.650 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:40:56.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:40:56.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:40:56.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=1 2022-04-10T01:40:56.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 1 = cluster1:1 ']' 2022-04-10T01:40:56.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:40:56.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:40:56.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=1 2022-04-10T01:40:56.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:40:56.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:40:56.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:40:56.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:56.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:40:56.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:1 cluster1 2022-04-10T01:40:56.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:1 2022-04-10T01:40:56.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:40:56.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:40:56.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2022-04-10T01:40:56.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:1 2022-04-10T01:40:56.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:40:56.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:40:56.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:40:56.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:40:56.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 1 = cluster1:1 ']' 2022-04-10T01:40:56.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:40:56.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:40:56.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=1 2022-04-10T01:40:56.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=1 2022-04-10T01:40:56.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:40:56.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:40:56.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:56.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:40:56.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2022-04-10T01:40:56.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:40:56.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:40:56.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:40:56.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_24 2022-04-10T01:40:56.769 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2022-04-10T01:40:56.769 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:40:56.769 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:40:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:40:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_24 2022-04-10T01:40:56.774 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:2 2022-04-10T01:40:56.774 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:40:56.774 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:40:56.774 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:2 cluster instance 2022-04-10T01:40:56.774 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:2 2022-04-10T01:40:56.775 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:40:56.775 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:40:56.775 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:40:56.775 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=2 2022-04-10T01:40:56.776 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 2 = cluster1:2 ']' 2022-04-10T01:40:56.776 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:40:56.776 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:40:56.776 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=2 2022-04-10T01:40:56.776 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:40:56.777 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:40:56.777 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:40:56.777 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:56.777 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:40:56.777 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:2 cluster1 2022-04-10T01:40:56.778 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:2 2022-04-10T01:40:56.778 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:40:56.778 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:40:56.778 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2022-04-10T01:40:56.779 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:2 2022-04-10T01:40:56.779 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:40:56.779 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:40:56.779 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:40:56.779 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:40:56.780 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 2 = cluster1:2 ']' 2022-04-10T01:40:56.780 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:40:56.780 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:40:56.780 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=2 2022-04-10T01:40:56.780 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=2 2022-04-10T01:40:56.781 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:40:56.781 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:40:56.781 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:56.781 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:40:56.781 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2022-04-10T01:40:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:40:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:40:56.788 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:40:56.788 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_24 2022-04-10T01:40:56.901 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:40:56.902 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:40:56.902 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:40:56.903 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:40:56.903 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:40:56.907 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:40:56.908 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:40:56.908 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:56.908 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_24 2022-04-10T01:40:56.908 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:40:56.909 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:56.909 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:56.909 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_24 2022-04-10T01:40:56.909 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:40:56.910 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:56.910 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:56.910 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:56.910 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:40:56.911 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:40:56.911 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_24 2022-04-10T01:40:56.911 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:40:56.911 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:40:56.911 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:56.912 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_24 2022-04-10T01:40:56.912 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:56.912 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:56.912 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:56.912 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:40:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-image_24.status 2022-04-10T01:40:56.914 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-image_24.status 2022-04-10T01:40:56.914 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:40:56.914 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_24.status 2022-04-10T01:40:56.914 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-image_24.status 2022-04-10T01:40:56.915 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-image_24.status 2022-04-10T01:40:56.915 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:40:56.915 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_24.status 2022-04-10T01:40:56.915 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror image_24 2022-04-10T01:40:56.916 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:56.916 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:56.916 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_24 2022-04-10T01:40:56.916 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:40:56.917 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:56.917 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:56.917 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:56.917 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_24 2022-04-10T01:40:57.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 188 2022-04-10T01:40:57.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_24 /tmp/tmp.rgwz74WspX/cluster2-mirror-image_24.status 2022-04-10T01:40:57.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:57.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:57.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_24 2022-04-10T01:40:57.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_24.status 2022-04-10T01:40:57.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:40:57.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:40:57.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:57.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:57.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_24 --format xml 2022-04-10T01:40:57.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:40:57.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:40:57.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=188 2022-04-10T01:40:57.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:40:57.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:40:57.566 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:40:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_24.status 2022-04-10T01:40:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_24 2022-04-10T01:40:57.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_24.status 2022-04-10T01:40:57.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:57.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:57.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:57.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:57.769 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2022-04-10T01:40:57.769 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:40:57.812 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:40:57.812 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:40:57.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:40:57.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:40:57.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:40:57.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:40:57.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 188 = '' 2022-04-10T01:40:57.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:40:57.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:40:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_24.status 2022-04-10T01:40:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_24 2022-04-10T01:40:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_24.status 2022-04-10T01:40:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2022-04-10T01:40:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:40:58.266 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:40:58.266 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:40:58.268 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:40:58.268 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:40:58.268 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:40:58.269 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:40:58.269 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 188 = '' 2022-04-10T01:40:58.269 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:40:58.269 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:40:59.071 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_24.status 2022-04-10T01:40:59.071 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:59.071 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:59.071 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_24 2022-04-10T01:40:59.072 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_24.status 2022-04-10T01:40:59.072 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:59.072 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:59.072 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:59.072 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:59.073 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2022-04-10T01:40:59.073 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:40:59.314 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:40:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=188 2022-04-10T01:40:59.317 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 188 = 188 2022-04-10T01:40:59.317 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:59.317 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror image_24 snap 2022-04-10T01:40:59.317 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:59.318 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:40:59.318 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_24 2022-04-10T01:40:59.318 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:40:59.318 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:40:59.318 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:40:59.319 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:40:59.319 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:40:59.319 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:40:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:40:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:41:00.317 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/image_24@snap 2022-04-10T01:41:00.360 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_24': 2022-04-10T01:41:00.360 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:41:00.361 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:41:00.361 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 2 2022-04-10T01:41:00.361 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 12a7a263037c 2022-04-10T01:41:00.361 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.12a7a263037c 2022-04-10T01:41:00.361 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:41:00.362 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:41:00.362 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:41:00.362 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:41:00.362 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:32:06 2022 2022-04-10T01:41:00.363 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:32:06 2022 2022-04-10T01:41:00.363 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:32:06 2022 2022-04-10T01:41:00.363 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:41:00.363 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:41:00.363 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:41:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: a52ba6c5-cf55-4fb1-99eb-fa7caff7c3bd 2022-04-10T01:41:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:41:00.365 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:00.365 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror image_24 snap 2022-04-10T01:41:00.365 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:00.366 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_24 2022-04-10T01:41:00.366 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:41:00.366 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:41:00.366 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_24.export 2022-04-10T01:41:00.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_24.export 2022-04-10T01:41:00.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_24.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_24.export 2022-04-10T01:41:00.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export image_24@snap /tmp/tmp.rgwz74WspX/cluster2-mirror-image_24.export 2022-04-10T01:41:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:41:00.556 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export image_24@snap /tmp/tmp.rgwz74WspX/cluster1-mirror-image_24.export 2022-04-10T01:41:00.793 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:41:00.797 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-image_24.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_24.export 2022-04-10T01:41:00.861 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_24.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_24.export 2022-04-10T01:41:00.898 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:00.898 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:41:00.899 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_25 2022-04-10T01:41:00.899 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror image_25 snap 2022-04-10T01:41:00.899 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:00.899 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:00.899 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_25 2022-04-10T01:41:00.900 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:41:00.900 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create image_25@snap --debug-rbd=20 --debug-journaler=20 2022-04-10T01:41:02.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror image_25 2022-04-10T01:41:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_25 2022-04-10T01:41:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:02.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:02.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:02.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:02.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror image_25 started 2022-04-10T01:41:02.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:02.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:02.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_25 2022-04-10T01:41:02.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:41:02.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:41:02.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:02.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:02.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:02.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:02.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:41:02.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:41:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror image_25 started 2022-04-10T01:41:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:03.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:03.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_25 2022-04-10T01:41:03.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:41:03.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:41:03.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:41:03.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:03.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:03.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:03.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:03.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_25 2022-04-10T01:41:03.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:41:03.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:41:03.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:41:03.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:41:03.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:41:03.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:41:03.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:41:03.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:41:03.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:03.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:41:03.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:41:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:41:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:41:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:41:03.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:41:03.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_25 2022-04-10T01:41:03.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:41:03.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:41:03.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:41:03.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:41:03.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:41:03.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:41:03.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:41:03.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:41:03.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:41:03.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:41:03.528 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:41:03.528 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:41:03.528 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:41:03.528 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:41:03.528 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:41:03.529 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:41:03.529 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:03.529 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:41:03.529 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:41:03.530 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:41:03.530 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:41:03.530 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:41:03.530 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:41:03.530 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:41:03.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:41:03.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:41:03.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:41:03.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:41:03.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:41:03.532 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:41:03.532 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:41:03.532 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:41:03.532 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:41:03.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:41:03.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:41:03.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:03.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:41:03.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:41:03.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:41:03.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:41:03.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:41:03.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_25 2022-04-10T01:41:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:41:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:41:03.661 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:41:03.661 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:41:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:41:03.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:41:03.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:41:03.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_25 2022-04-10T01:41:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:41:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_25 2022-04-10T01:41:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:41:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:41:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:41:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_25 2022-04-10T01:41:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:41:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:41:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:03.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_25 2022-04-10T01:41:03.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:03.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:03.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:03.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:03.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:41:03.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:03.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:03.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:03.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-image_25.status 2022-04-10T01:41:03.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-image_25.status 2022-04-10T01:41:03.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:41:03.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_25.status 2022-04-10T01:41:03.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-image_25.status 2022-04-10T01:41:03.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-image_25.status 2022-04-10T01:41:03.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:41:03.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_25.status 2022-04-10T01:41:03.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror image_25 2022-04-10T01:41:03.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:03.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:03.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_25 2022-04-10T01:41:03.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:41:03.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:03.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:03.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:03.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_25 2022-04-10T01:41:04.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 191 2022-04-10T01:41:04.530 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_25 /tmp/tmp.rgwz74WspX/cluster2-mirror-image_25.status 2022-04-10T01:41:04.530 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:04.530 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:04.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_25 2022-04-10T01:41:04.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_25.status 2022-04-10T01:41:04.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:41:04.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:04.532 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:04.532 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:04.532 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_25 --format xml 2022-04-10T01:41:04.532 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:41:04.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:41:04.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=191 2022-04-10T01:41:04.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:41:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:41:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:41:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_25.status 2022-04-10T01:41:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_25 2022-04-10T01:41:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_25.status 2022-04-10T01:41:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2022-04-10T01:41:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:41:04.823 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:41:04.824 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:41:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:41:04.826 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:41:04.826 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:41:04.827 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:41:04.827 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 191 = '' 2022-04-10T01:41:04.827 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:41:04.827 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:41:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_25.status 2022-04-10T01:41:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:05.229 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_25 2022-04-10T01:41:05.229 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_25.status 2022-04-10T01:41:05.229 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:05.229 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:05.230 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:05.230 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:05.230 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2022-04-10T01:41:05.230 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:41:05.275 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:41:05.276 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:41:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:41:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:41:05.278 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:41:05.278 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:41:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 191 = '' 2022-04-10T01:41:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:41:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:41:06.080 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_25.status 2022-04-10T01:41:06.080 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:06.080 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:06.080 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_25 2022-04-10T01:41:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_25.status 2022-04-10T01:41:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2022-04-10T01:41:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:41:06.121 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:41:06.124 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=191 2022-04-10T01:41:06.124 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 191 = 191 2022-04-10T01:41:06.124 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:06.125 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror image_25 snap 2022-04-10T01:41:06.125 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:06.125 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:06.125 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_25 2022-04-10T01:41:06.125 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:41:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:41:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:06.127 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:41:06.127 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:41:07.124 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/image_25@snap 2022-04-10T01:41:07.167 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_25': 2022-04-10T01:41:07.168 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:41:07.168 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:41:07.168 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 2 2022-04-10T01:41:07.168 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 10e44c4ea710 2022-04-10T01:41:07.169 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.10e44c4ea710 2022-04-10T01:41:07.169 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:41:07.169 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:41:07.169 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:41:07.170 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:41:07.170 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:32:18 2022 2022-04-10T01:41:07.170 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:32:18 2022 2022-04-10T01:41:07.170 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:32:18 2022 2022-04-10T01:41:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:41:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:41:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:41:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: a8a88faf-cd63-4557-9afd-ca8d6505edb6 2022-04-10T01:41:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:41:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror image_25 snap 2022-04-10T01:41:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_25 2022-04-10T01:41:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:41:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:41:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_25.export 2022-04-10T01:41:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_25.export 2022-04-10T01:41:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_25.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_25.export 2022-04-10T01:41:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export image_25@snap /tmp/tmp.rgwz74WspX/cluster2-mirror-image_25.export 2022-04-10T01:41:07.350 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:41:07.353 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export image_25@snap /tmp/tmp.rgwz74WspX/cluster1-mirror-image_25.export 2022-04-10T01:41:07.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:41:07.602 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-image_25.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_25.export 2022-04-10T01:41:07.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_25.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_25.export 2022-04-10T01:41:07.704 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:07.704 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:41:07.704 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_26 2022-04-10T01:41:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror image_26 snap 2022-04-10T01:41:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_26 2022-04-10T01:41:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:41:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create image_26@snap --debug-rbd=20 --debug-journaler=20 2022-04-10T01:41:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror image_26 2022-04-10T01:41:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_26 2022-04-10T01:41:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror image_26 started 2022-04-10T01:41:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:09.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_26 2022-04-10T01:41:09.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:41:09.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:41:09.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:09.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:09.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:09.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:09.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:41:09.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:41:10.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror image_26 started 2022-04-10T01:41:10.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:10.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:10.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_26 2022-04-10T01:41:10.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:41:10.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:41:10.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:41:10.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:10.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:10.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:10.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:10.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_26 2022-04-10T01:41:10.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:41:10.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:41:10.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:41:10.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:41:10.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:41:10.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:41:10.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:41:10.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:41:10.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:10.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:41:10.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:41:10.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:41:10.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:41:10.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:41:10.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:41:10.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_26 2022-04-10T01:41:10.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:41:10.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:41:10.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:41:10.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:41:10.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:41:10.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:41:10.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:41:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:41:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:41:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:41:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:41:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:41:10.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:41:10.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:41:10.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:41:10.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:41:10.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:10.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:41:10.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:41:10.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:41:10.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:41:10.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:41:10.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:41:10.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:41:10.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:41:10.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:41:10.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:41:10.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:41:10.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:41:10.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:41:10.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:41:10.528 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:41:10.528 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:41:10.528 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:41:10.528 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:41:10.528 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:10.529 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:41:10.529 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:41:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:41:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:41:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:41:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_26 2022-04-10T01:41:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:41:10.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:41:10.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:41:10.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:41:10.658 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:41:10.660 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:41:10.661 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:41:10.661 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:10.661 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_26 2022-04-10T01:41:10.661 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:41:10.662 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:10.662 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:10.662 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_26 2022-04-10T01:41:10.662 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:41:10.662 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:10.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:10.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:10.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:41:10.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:41:10.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_26 2022-04-10T01:41:10.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:41:10.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:41:10.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:10.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_26 2022-04-10T01:41:10.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:10.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:10.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:10.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:10.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:41:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-image_26.status 2022-04-10T01:41:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-image_26.status 2022-04-10T01:41:10.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:41:10.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_26.status 2022-04-10T01:41:10.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-image_26.status 2022-04-10T01:41:10.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-image_26.status 2022-04-10T01:41:10.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:41:10.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_26.status 2022-04-10T01:41:10.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror image_26 2022-04-10T01:41:10.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:10.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_26 2022-04-10T01:41:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:41:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_26 2022-04-10T01:41:11.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 194 2022-04-10T01:41:11.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_26 /tmp/tmp.rgwz74WspX/cluster2-mirror-image_26.status 2022-04-10T01:41:11.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:11.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:11.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_26 2022-04-10T01:41:11.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_26.status 2022-04-10T01:41:11.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:41:11.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:11.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:11.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:11.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_26 --format xml 2022-04-10T01:41:11.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:41:11.592 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:41:11.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=194 2022-04-10T01:41:11.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:41:11.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:41:11.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:41:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_26.status 2022-04-10T01:41:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:11.797 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_26 2022-04-10T01:41:11.797 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_26.status 2022-04-10T01:41:11.797 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:11.797 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:11.798 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:11.798 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:11.798 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2022-04-10T01:41:11.798 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:41:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:41:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:41:11.844 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:41:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:41:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:41:11.846 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:41:11.846 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 194 = '' 2022-04-10T01:41:11.846 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:41:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:41:12.247 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_26.status 2022-04-10T01:41:12.248 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:12.248 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:12.248 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_26 2022-04-10T01:41:12.248 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_26.status 2022-04-10T01:41:12.248 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:12.249 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:12.249 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:12.249 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:12.249 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2022-04-10T01:41:12.249 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:41:12.295 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:41:12.295 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:41:12.297 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:41:12.297 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:41:12.297 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:41:12.298 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:41:12.298 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 194 = '' 2022-04-10T01:41:12.298 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:41:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:41:13.100 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_26.status 2022-04-10T01:41:13.100 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:13.100 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:13.100 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_26 2022-04-10T01:41:13.101 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_26.status 2022-04-10T01:41:13.101 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:13.101 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:13.101 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:13.101 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2022-04-10T01:41:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:41:13.140 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:41:13.143 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=194 2022-04-10T01:41:13.144 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 194 = 194 2022-04-10T01:41:13.144 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:13.144 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror image_26 snap 2022-04-10T01:41:13.144 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:13.144 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:13.145 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_26 2022-04-10T01:41:13.145 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:41:13.145 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:41:13.145 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:13.145 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:13.146 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:13.146 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:13.146 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:41:13.146 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:41:13.184 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.665748534122676) > weight (0.25) 2022-04-10T01:41:13.184 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2022-04-10T01:41:13.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2022-04-10T01:41:13.185 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2022-04-10T01:41:13.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi043.stderr:2022-04-10T01:41:13.184+0000 7f3bc073d700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 58058) UID: 1000 2022-04-10T01:41:13.386 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi043.stderr:daemon-helper: command failed with exit status 1 2022-04-10T01:41:14.144 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/image_26@snap 2022-04-10T01:41:14.185 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2022-04-10T01:41:14.186 INFO:teuthology.orchestra.run:waiting for 600 2022-04-10T01:41:14.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-04-10T01:41:14.187 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2022-04-10T01:41:14.187 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2022-04-10T01:41:14.187 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2022-04-10T01:41:14.188 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_26': 2022-04-10T01:41:14.188 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:41:14.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:41:14.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 2 2022-04-10T01:41:14.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 11331e3ac486 2022-04-10T01:41:14.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.11331e3ac486 2022-04-10T01:41:14.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:41:14.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:41:14.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:41:14.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:41:14.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:32:32 2022 2022-04-10T01:41:14.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:32:32 2022 2022-04-10T01:41:14.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:32:32 2022 2022-04-10T01:41:14.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:41:14.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:41:14.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:41:14.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: ef9f36c8-5f7f-4c73-8f03-3933b20566c7 2022-04-10T01:41:14.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:41:14.193 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:14.193 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror image_26 snap 2022-04-10T01:41:14.194 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:14.194 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_26 2022-04-10T01:41:14.194 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:41:14.194 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:41:14.195 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_26.export 2022-04-10T01:41:14.195 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_26.export 2022-04-10T01:41:14.195 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_26.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_26.export 2022-04-10T01:41:14.195 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export image_26@snap /tmp/tmp.rgwz74WspX/cluster2-mirror-image_26.export 2022-04-10T01:41:14.196 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2022-04-10T01:41:14.196 INFO:tasks.rbd_mirror_thrash:waiting for 75 secs before thrashing 2022-04-10T01:41:14.374 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:41:14.378 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export image_26@snap /tmp/tmp.rgwz74WspX/cluster1-mirror-image_26.export 2022-04-10T01:41:14.650 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:41:14.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-image_26.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_26.export 2022-04-10T01:41:14.717 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_26.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_26.export 2022-04-10T01:41:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:41:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_27 2022-04-10T01:41:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror image_27 snap 2022-04-10T01:41:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:14.757 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_27 2022-04-10T01:41:14.757 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:41:14.757 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create image_27@snap --debug-rbd=20 --debug-journaler=20 2022-04-10T01:41:16.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror image_27 2022-04-10T01:41:16.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:16.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:16.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_27 2022-04-10T01:41:16.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:16.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:16.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:16.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:16.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror image_27 started 2022-04-10T01:41:16.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:16.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:16.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_27 2022-04-10T01:41:16.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:41:16.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:41:16.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:16.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:16.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:16.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:16.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:41:16.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:41:17.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror image_27 started 2022-04-10T01:41:17.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:17.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:17.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_27 2022-04-10T01:41:17.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:41:17.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:41:17.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:41:17.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:17.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:17.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:17.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:17.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_27 2022-04-10T01:41:17.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:41:17.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:41:17.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:41:17.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:41:17.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:41:17.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:41:17.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:41:17.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:41:17.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:17.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:41:17.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:41:17.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:41:17.557 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:41:17.557 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:41:17.557 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:41:17.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2022-04-10T01:41:17.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:41:17.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:41:17.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:41:17.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:41:17.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:41:17.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:41:17.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:41:17.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:41:17.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:41:17.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:41:17.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:41:17.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:41:17.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:41:17.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:41:17.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:41:17.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:41:17.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:17.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:41:17.562 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:41:17.562 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:41:17.562 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:41:17.562 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:41:17.562 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:41:17.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:41:17.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:41:17.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:41:17.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:41:17.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:41:17.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:41:17.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:41:17.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:41:17.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:41:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:41:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:41:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:41:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:41:17.566 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:41:17.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:41:17.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:41:17.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:41:17.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2022-04-10T01:41:17.699 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:41:17.699 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:41:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:41:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:41:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:41:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:41:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:41:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_27 2022-04-10T01:41:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:41:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_27 2022-04-10T01:41:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:41:17.707 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:17.707 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:17.707 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:17.707 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:41:17.707 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:41:17.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_27 2022-04-10T01:41:17.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:41:17.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:41:17.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:17.709 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_27 2022-04-10T01:41:17.709 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:17.709 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:17.709 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:17.709 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:41:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-image_27.status 2022-04-10T01:41:17.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-image_27.status 2022-04-10T01:41:17.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:41:17.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_27.status 2022-04-10T01:41:17.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-image_27.status 2022-04-10T01:41:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-image_27.status 2022-04-10T01:41:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:41:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_27.status 2022-04-10T01:41:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror image_27 2022-04-10T01:41:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_27 2022-04-10T01:41:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:41:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:17.714 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_27 2022-04-10T01:41:18.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 197 2022-04-10T01:41:18.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_27 /tmp/tmp.rgwz74WspX/cluster2-mirror-image_27.status 2022-04-10T01:41:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_27 2022-04-10T01:41:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_27.status 2022-04-10T01:41:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:41:18.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:18.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:18.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:18.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_27 --format xml 2022-04-10T01:41:18.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:41:18.607 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:41:18.609 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=197 2022-04-10T01:41:18.609 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:41:18.609 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:41:18.610 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:41:18.811 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_27.status 2022-04-10T01:41:18.811 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_27 2022-04-10T01:41:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_27.status 2022-04-10T01:41:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2022-04-10T01:41:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:41:18.863 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:41:18.864 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:41:18.865 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:41:18.865 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:41:18.866 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:41:18.867 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:41:18.867 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 197 = '' 2022-04-10T01:41:18.867 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:41:18.867 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:41:19.268 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_27.status 2022-04-10T01:41:19.268 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:19.268 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:19.269 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_27 2022-04-10T01:41:19.269 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_27.status 2022-04-10T01:41:19.269 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:19.269 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:19.269 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:19.270 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:19.270 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2022-04-10T01:41:19.270 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:41:19.315 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:41:19.316 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:41:19.317 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:41:19.317 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:41:19.318 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:41:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:41:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 197 = '' 2022-04-10T01:41:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:41:19.321 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:41:20.119 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_27.status 2022-04-10T01:41:20.120 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:20.120 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:20.120 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_27 2022-04-10T01:41:20.121 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_27.status 2022-04-10T01:41:20.121 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:20.121 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:20.121 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:20.122 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:20.122 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2022-04-10T01:41:20.122 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:41:20.160 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:41:20.163 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=197 2022-04-10T01:41:20.163 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 197 = 197 2022-04-10T01:41:20.164 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:20.164 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror image_27 snap 2022-04-10T01:41:20.164 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:20.164 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_27 2022-04-10T01:41:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:41:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:41:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:20.166 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:20.166 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:20.166 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:41:20.166 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:41:21.163 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/image_27@snap 2022-04-10T01:41:21.208 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_27': 2022-04-10T01:41:21.208 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:41:21.208 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:41:21.208 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 2 2022-04-10T01:41:21.209 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 118daf03a1fa 2022-04-10T01:41:21.209 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.118daf03a1fa 2022-04-10T01:41:21.209 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:41:21.209 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:41:21.209 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:41:21.210 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:41:21.210 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:32:46 2022 2022-04-10T01:41:21.210 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:32:46 2022 2022-04-10T01:41:21.210 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:32:46 2022 2022-04-10T01:41:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:41:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:41:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:41:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 5e030d77-6c1f-4e4a-a414-67687916c7e7 2022-04-10T01:41:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:41:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror image_27 snap 2022-04-10T01:41:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_27 2022-04-10T01:41:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:41:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:41:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_27.export 2022-04-10T01:41:21.215 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_27.export 2022-04-10T01:41:21.215 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_27.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_27.export 2022-04-10T01:41:21.215 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export image_27@snap /tmp/tmp.rgwz74WspX/cluster2-mirror-image_27.export 2022-04-10T01:41:21.398 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:41:21.402 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export image_27@snap /tmp/tmp.rgwz74WspX/cluster1-mirror-image_27.export 2022-04-10T01:41:21.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:41:21.628 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-image_27.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_27.export 2022-04-10T01:41:21.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_27.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_27.export 2022-04-10T01:41:21.729 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:21.730 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:41:21.730 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_28 2022-04-10T01:41:21.730 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror image_28 snap 2022-04-10T01:41:21.730 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:21.731 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:21.731 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_28 2022-04-10T01:41:21.731 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:41:21.731 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create image_28@snap --debug-rbd=20 --debug-journaler=20 2022-04-10T01:41:23.567 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror image_28 2022-04-10T01:41:23.567 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:23.567 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:23.568 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_28 2022-04-10T01:41:23.568 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:23.568 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:23.568 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:23.568 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:23.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror image_28 started 2022-04-10T01:41:23.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:23.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:23.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_28 2022-04-10T01:41:23.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:41:23.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:41:23.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:23.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:23.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:23.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:23.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:41:23.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:41:24.568 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror image_28 started 2022-04-10T01:41:24.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:24.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:24.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_28 2022-04-10T01:41:24.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:41:24.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:41:24.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:41:24.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:24.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_28 2022-04-10T01:41:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:41:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:41:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:41:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:41:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:41:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:41:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:41:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:41:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:41:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:41:24.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:41:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:41:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:41:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:41:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_28 2022-04-10T01:41:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:41:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:41:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:41:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:41:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:41:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:41:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:41:24.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:41:24.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:41:24.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:41:24.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:41:24.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:41:24.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:41:24.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:41:24.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:41:24.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:41:24.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:24.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:41:24.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:41:24.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:41:24.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:41:24.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:41:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:41:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:41:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:41:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:41:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:41:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:41:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:41:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:41:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:41:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:41:24.585 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:41:24.585 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:41:24.585 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:41:24.585 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:24.585 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:41:24.586 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:41:24.591 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:41:24.591 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:41:24.592 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:41:24.592 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_28 2022-04-10T01:41:24.704 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2022-04-10T01:41:24.704 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:41:24.705 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:41:24.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:41:24.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_28 2022-04-10T01:41:24.709 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:1 2022-04-10T01:41:24.709 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:41:24.709 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:41:24.709 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:1 cluster instance 2022-04-10T01:41:24.709 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:1 2022-04-10T01:41:24.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:41:24.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:41:24.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:41:24.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=1 2022-04-10T01:41:24.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 1 = cluster1:1 ']' 2022-04-10T01:41:24.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:41:24.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:41:24.712 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=1 2022-04-10T01:41:24.712 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:41:24.712 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:41:24.712 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:41:24.712 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:24.713 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:41:24.713 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:1 cluster1 2022-04-10T01:41:24.713 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:1 2022-04-10T01:41:24.713 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:41:24.713 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:41:24.714 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2022-04-10T01:41:24.714 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:1 2022-04-10T01:41:24.714 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:41:24.714 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:41:24.714 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:41:24.715 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:41:24.715 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 1 = cluster1:1 ']' 2022-04-10T01:41:24.715 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:41:24.715 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:41:24.715 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=1 2022-04-10T01:41:24.716 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=1 2022-04-10T01:41:24.716 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:41:24.716 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:41:24.716 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:24.716 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:41:24.717 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2022-04-10T01:41:24.723 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:41:24.724 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:41:24.724 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:41:24.724 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_28 2022-04-10T01:41:24.829 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2022-04-10T01:41:24.829 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:41:24.829 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:41:24.833 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:41:24.833 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_28 2022-04-10T01:41:24.833 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:2 2022-04-10T01:41:24.833 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:41:24.834 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:41:24.834 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:2 cluster instance 2022-04-10T01:41:24.834 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:2 2022-04-10T01:41:24.834 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:41:24.834 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:41:24.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:41:24.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=2 2022-04-10T01:41:24.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 2 = cluster1:2 ']' 2022-04-10T01:41:24.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:41:24.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:41:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=2 2022-04-10T01:41:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:41:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:41:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:41:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:41:24.837 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:2 cluster1 2022-04-10T01:41:24.837 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:2 2022-04-10T01:41:24.837 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:41:24.837 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:41:24.837 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2022-04-10T01:41:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:2 2022-04-10T01:41:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:41:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:41:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:41:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:41:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 2 = cluster1:2 ']' 2022-04-10T01:41:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:41:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:41:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=2 2022-04-10T01:41:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=2 2022-04-10T01:41:24.840 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:41:24.840 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:41:24.840 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:24.840 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:41:24.840 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2022-04-10T01:41:24.847 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:41:24.848 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:41:24.848 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:41:24.848 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_28 2022-04-10T01:41:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2022-04-10T01:41:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:41:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:41:24.958 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:41:24.958 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_28 2022-04-10T01:41:24.958 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:3 2022-04-10T01:41:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:41:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:41:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:3 cluster instance 2022-04-10T01:41:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:3 2022-04-10T01:41:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:41:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:41:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:41:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=3 2022-04-10T01:41:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 3 = cluster1:3 ']' 2022-04-10T01:41:24.961 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:41:24.961 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:41:24.961 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=3 2022-04-10T01:41:24.961 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=3 2022-04-10T01:41:24.961 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:41:24.961 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:41:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:41:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:3 cluster1 2022-04-10T01:41:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:3 2022-04-10T01:41:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:41:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:41:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2022-04-10T01:41:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:3 2022-04-10T01:41:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:41:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:41:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:41:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=3 2022-04-10T01:41:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 3 = cluster1:3 ']' 2022-04-10T01:41:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:41:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:41:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=3 2022-04-10T01:41:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=3 2022-04-10T01:41:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:41:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:41:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:41:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2022-04-10T01:41:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:41:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:41:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:41:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_28 2022-04-10T01:41:25.088 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:41:25.089 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:41:25.089 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:41:25.090 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:41:25.090 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:41:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:41:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:41:25.094 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:25.094 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_28 2022-04-10T01:41:25.094 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:41:25.094 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_28 2022-04-10T01:41:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:41:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:25.096 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:41:25.096 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:41:25.096 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_28 2022-04-10T01:41:25.096 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:41:25.096 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:41:25.097 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:25.097 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_28 2022-04-10T01:41:25.097 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:25.097 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:41:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-image_28.status 2022-04-10T01:41:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-image_28.status 2022-04-10T01:41:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:41:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_28.status 2022-04-10T01:41:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-image_28.status 2022-04-10T01:41:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-image_28.status 2022-04-10T01:41:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:41:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_28.status 2022-04-10T01:41:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror image_28 2022-04-10T01:41:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_28 2022-04-10T01:41:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:41:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_28 2022-04-10T01:41:25.566 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 200 2022-04-10T01:41:25.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_28 /tmp/tmp.rgwz74WspX/cluster2-mirror-image_28.status 2022-04-10T01:41:25.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:25.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:25.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_28 2022-04-10T01:41:25.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_28.status 2022-04-10T01:41:25.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:41:25.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:25.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_28 --format xml 2022-04-10T01:41:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:41:25.616 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:41:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=200 2022-04-10T01:41:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:41:25.722 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:41:25.722 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:41:25.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_28.status 2022-04-10T01:41:25.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:25.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:25.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_28 2022-04-10T01:41:25.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_28.status 2022-04-10T01:41:25.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:25.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:25.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:25.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:25.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2022-04-10T01:41:25.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:41:25.862 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:41:25.862 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:41:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:41:25.864 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:41:25.864 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:41:25.865 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:41:25.865 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 200 = '' 2022-04-10T01:41:25.865 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:41:25.865 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:41:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_28.status 2022-04-10T01:41:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_28 2022-04-10T01:41:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_28.status 2022-04-10T01:41:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2022-04-10T01:41:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:41:26.314 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:41:26.314 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:41:26.316 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:41:26.316 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:41:26.317 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:41:26.317 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:41:26.317 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 200 = '' 2022-04-10T01:41:26.317 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:41:26.318 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:41:27.118 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_28.status 2022-04-10T01:41:27.118 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:27.119 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:27.119 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_28 2022-04-10T01:41:27.119 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_28.status 2022-04-10T01:41:27.119 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:27.120 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:27.120 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:27.120 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:27.120 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2022-04-10T01:41:27.120 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:41:27.159 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:41:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=200 2022-04-10T01:41:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 200 = 200 2022-04-10T01:41:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:27.163 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror image_28 snap 2022-04-10T01:41:27.163 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:27.163 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:27.163 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_28 2022-04-10T01:41:27.163 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:41:27.164 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:41:27.164 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:27.164 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:27.164 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:27.164 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:41:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:41:28.162 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/image_28@snap 2022-04-10T01:41:28.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_28': 2022-04-10T01:41:28.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:41:28.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:41:28.207 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 2 2022-04-10T01:41:28.207 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 12a77b2e0633 2022-04-10T01:41:28.207 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.12a77b2e0633 2022-04-10T01:41:28.207 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:41:28.208 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:41:28.208 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:41:28.208 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:41:28.208 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:32:56 2022 2022-04-10T01:41:28.209 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:32:56 2022 2022-04-10T01:41:28.209 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:32:56 2022 2022-04-10T01:41:28.209 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:41:28.209 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:41:28.209 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:41:28.210 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: e100174d-382f-42fc-9a30-2ef4065b1b6a 2022-04-10T01:41:28.210 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:41:28.211 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:28.211 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror image_28 snap 2022-04-10T01:41:28.212 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:28.212 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_28 2022-04-10T01:41:28.212 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:41:28.212 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:41:28.213 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_28.export 2022-04-10T01:41:28.213 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_28.export 2022-04-10T01:41:28.213 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_28.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_28.export 2022-04-10T01:41:28.213 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export image_28@snap /tmp/tmp.rgwz74WspX/cluster2-mirror-image_28.export 2022-04-10T01:41:28.407 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:41:28.410 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export image_28@snap /tmp/tmp.rgwz74WspX/cluster1-mirror-image_28.export 2022-04-10T01:41:28.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:41:28.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-image_28.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_28.export 2022-04-10T01:41:28.735 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_28.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_28.export 2022-04-10T01:41:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:41:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_29 2022-04-10T01:41:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror image_29 snap 2022-04-10T01:41:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:28.774 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:28.774 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_29 2022-04-10T01:41:28.774 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:41:28.774 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create image_29@snap --debug-rbd=20 --debug-journaler=20 2022-04-10T01:41:30.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror image_29 2022-04-10T01:41:30.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:30.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:30.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_29 2022-04-10T01:41:30.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:30.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:30.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:30.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:30.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror image_29 started 2022-04-10T01:41:30.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:30.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:30.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_29 2022-04-10T01:41:30.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:41:30.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:41:30.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:41:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:41:31.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror image_29 started 2022-04-10T01:41:31.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:31.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_29 2022-04-10T01:41:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:41:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:41:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:41:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_29 2022-04-10T01:41:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:41:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:41:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:41:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:41:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:41:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:41:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:41:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:41:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:31.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:41:31.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:41:31.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:41:31.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:41:31.528 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:41:31.528 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:41:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_29 2022-04-10T01:41:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:41:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:41:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:41:31.530 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:41:31.530 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:41:31.530 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:41:31.530 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:41:31.530 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:41:31.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:41:31.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:41:31.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:41:31.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:41:31.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:41:31.532 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:41:31.532 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:41:31.532 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:41:31.532 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:31.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:41:31.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:41:31.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:41:31.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:41:31.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:41:31.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:41:31.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:41:31.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:41:31.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:41:31.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:41:31.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:41:31.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:41:31.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:41:31.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:41:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:41:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:41:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:41:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:41:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:31.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:41:31.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:41:31.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:41:31.541 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:41:31.541 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:41:31.541 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_29 2022-04-10T01:41:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:41:31.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:41:31.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:41:31.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:41:31.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:41:31.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:41:31.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:41:31.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:31.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_29 2022-04-10T01:41:31.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:41:31.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:31.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:31.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_29 2022-04-10T01:41:31.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:41:31.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:31.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:31.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:31.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:41:31.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:41:31.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_29 2022-04-10T01:41:31.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:41:31.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:41:31.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:31.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_29 2022-04-10T01:41:31.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:31.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:31.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:31.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:31.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:41:31.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:31.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:31.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:31.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-image_29.status 2022-04-10T01:41:31.677 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-image_29.status 2022-04-10T01:41:31.677 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:41:31.677 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_29.status 2022-04-10T01:41:31.677 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-image_29.status 2022-04-10T01:41:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-image_29.status 2022-04-10T01:41:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:41:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_29.status 2022-04-10T01:41:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror image_29 2022-04-10T01:41:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:31.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:31.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_29 2022-04-10T01:41:31.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:41:31.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:31.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:31.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:31.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_29 2022-04-10T01:41:32.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 203 2022-04-10T01:41:32.603 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_29 /tmp/tmp.rgwz74WspX/cluster2-mirror-image_29.status 2022-04-10T01:41:32.603 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:32.603 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:32.604 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_29 2022-04-10T01:41:32.604 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_29.status 2022-04-10T01:41:32.604 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:41:32.604 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:32.605 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:32.605 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:32.605 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_29 --format xml 2022-04-10T01:41:32.605 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:41:32.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:41:32.647 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=203 2022-04-10T01:41:32.648 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:41:32.648 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:41:32.648 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:41:32.849 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_29.status 2022-04-10T01:41:32.850 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:32.850 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:32.850 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_29 2022-04-10T01:41:32.850 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_29.status 2022-04-10T01:41:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2022-04-10T01:41:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:41:32.895 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:41:32.896 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:41:32.897 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:41:32.897 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:41:32.898 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:41:32.899 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:41:32.899 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 203 = '' 2022-04-10T01:41:32.899 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:41:32.899 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:41:33.300 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_29.status 2022-04-10T01:41:33.301 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:33.301 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:33.301 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_29 2022-04-10T01:41:33.301 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_29.status 2022-04-10T01:41:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2022-04-10T01:41:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:41:33.347 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:41:33.347 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:41:33.349 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:41:33.349 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:41:33.349 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:41:33.350 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:41:33.350 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 203 = '' 2022-04-10T01:41:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:41:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:41:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_29.status 2022-04-10T01:41:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_29 2022-04-10T01:41:34.153 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_29.status 2022-04-10T01:41:34.153 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:34.153 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:34.153 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:34.154 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:34.154 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2022-04-10T01:41:34.154 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:41:34.193 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:41:34.196 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=203 2022-04-10T01:41:34.197 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 203 = 203 2022-04-10T01:41:34.197 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:34.197 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror image_29 snap 2022-04-10T01:41:34.197 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:34.197 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_29 2022-04-10T01:41:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:41:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:41:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:34.199 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:34.199 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:34.199 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:41:34.199 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:41:35.197 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/image_29@snap 2022-04-10T01:41:35.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_29': 2022-04-10T01:41:35.242 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:41:35.242 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:41:35.242 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 2 2022-04-10T01:41:35.243 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 10e4fd2c8a22 2022-04-10T01:41:35.243 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.10e4fd2c8a22 2022-04-10T01:41:35.243 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:41:35.243 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:41:35.243 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:41:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:41:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:33:10 2022 2022-04-10T01:41:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:33:10 2022 2022-04-10T01:41:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:33:10 2022 2022-04-10T01:41:35.245 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:41:35.245 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:41:35.245 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:41:35.245 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 617e30a5-cc17-4271-95fb-ba4832408157 2022-04-10T01:41:35.245 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:41:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror image_29 snap 2022-04-10T01:41:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_29 2022-04-10T01:41:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:41:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:41:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_29.export 2022-04-10T01:41:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_29.export 2022-04-10T01:41:35.249 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_29.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_29.export 2022-04-10T01:41:35.249 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export image_29@snap /tmp/tmp.rgwz74WspX/cluster2-mirror-image_29.export 2022-04-10T01:41:35.430 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:41:35.433 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export image_29@snap /tmp/tmp.rgwz74WspX/cluster1-mirror-image_29.export 2022-04-10T01:41:35.659 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:41:35.661 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-image_29.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_29.export 2022-04-10T01:41:35.725 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_29.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_29.export 2022-04-10T01:41:35.763 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:35.763 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:41:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_30 2022-04-10T01:41:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror image_30 snap 2022-04-10T01:41:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:35.765 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_30 2022-04-10T01:41:35.765 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:41:35.765 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create image_30@snap --debug-rbd=20 --debug-journaler=20 2022-04-10T01:41:37.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror image_30 2022-04-10T01:41:37.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:37.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:37.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_30 2022-04-10T01:41:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror image_30 started 2022-04-10T01:41:37.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:37.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:37.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_30 2022-04-10T01:41:37.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:41:37.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:41:37.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:37.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:37.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:37.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:37.600 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:41:37.600 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:41:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror image_30 started 2022-04-10T01:41:38.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:38.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:38.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_30 2022-04-10T01:41:38.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:41:38.600 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:41:38.600 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:41:38.600 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:38.600 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:38.600 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_30 2022-04-10T01:41:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:41:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:41:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:41:38.602 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:41:38.602 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:41:38.602 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:41:38.602 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:41:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:41:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:41:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:41:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:41:38.604 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:41:38.604 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:41:38.608 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:41:38.608 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2022-04-10T01:41:38.608 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:41:38.608 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:41:38.609 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:41:38.609 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:41:38.609 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:41:38.609 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:41:38.610 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:41:38.610 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:41:38.610 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:41:38.610 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:41:38.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:41:38.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:41:38.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:41:38.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:41:38.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:41:38.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:41:38.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:38.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:41:38.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:41:38.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:41:38.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:41:38.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:41:38.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:41:38.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:41:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:41:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:41:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:41:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:41:38.615 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:41:38.615 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:41:38.615 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:41:38.615 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:41:38.615 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:41:38.616 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:41:38.616 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:41:38.616 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:38.616 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:41:38.617 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:41:38.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:41:38.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:41:38.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:41:38.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2022-04-10T01:41:38.743 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:41:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:41:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:41:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:41:38.745 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:41:38.748 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:41:38.748 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:41:38.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:38.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_30 2022-04-10T01:41:38.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:41:38.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:38.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:38.750 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_30 2022-04-10T01:41:38.750 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:41:38.750 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:38.750 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:38.751 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:38.751 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:41:38.751 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:41:38.751 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_30 2022-04-10T01:41:38.751 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:41:38.752 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:41:38.752 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:38.752 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_30 2022-04-10T01:41:38.752 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:38.752 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:38.753 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:38.753 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:38.753 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:41:38.753 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:38.753 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:38.754 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:38.754 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-image_30.status 2022-04-10T01:41:38.754 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-image_30.status 2022-04-10T01:41:38.754 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:41:38.754 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_30.status 2022-04-10T01:41:38.755 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-image_30.status 2022-04-10T01:41:38.755 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-image_30.status 2022-04-10T01:41:38.755 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:41:38.756 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_30.status 2022-04-10T01:41:38.757 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror image_30 2022-04-10T01:41:38.757 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:38.757 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:38.757 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_30 2022-04-10T01:41:38.757 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:41:38.758 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:38.758 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:38.758 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:38.758 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_30 2022-04-10T01:41:39.601 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 206 2022-04-10T01:41:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_30 /tmp/tmp.rgwz74WspX/cluster2-mirror-image_30.status 2022-04-10T01:41:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:39.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_30 2022-04-10T01:41:39.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_30.status 2022-04-10T01:41:39.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:41:39.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:39.614 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:39.614 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:39.614 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_30 --format xml 2022-04-10T01:41:39.614 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:41:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:41:39.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=206 2022-04-10T01:41:39.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:41:39.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:41:39.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:41:39.858 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_30.status 2022-04-10T01:41:39.859 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:39.859 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:39.859 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_30 2022-04-10T01:41:39.860 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_30.status 2022-04-10T01:41:39.860 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:39.860 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:39.860 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:39.860 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:39.861 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2022-04-10T01:41:39.861 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:41:39.904 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:41:39.904 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:41:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:41:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:41:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:41:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:41:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 206 = '' 2022-04-10T01:41:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:41:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:41:40.308 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_30.status 2022-04-10T01:41:40.308 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:40.308 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:40.309 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_30 2022-04-10T01:41:40.309 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_30.status 2022-04-10T01:41:40.309 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:40.309 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:40.309 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:40.310 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:40.310 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2022-04-10T01:41:40.310 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:41:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:41:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:41:40.357 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:41:40.357 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:41:40.358 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:41:40.358 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:41:40.359 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 206 = '' 2022-04-10T01:41:40.359 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:41:40.359 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:41:41.160 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_30.status 2022-04-10T01:41:41.160 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:41.161 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:41.161 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_30 2022-04-10T01:41:41.161 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_30.status 2022-04-10T01:41:41.161 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:41.161 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:41.162 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:41.162 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:41.162 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2022-04-10T01:41:41.162 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:41:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:41:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=206 2022-04-10T01:41:41.408 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 206 = 206 2022-04-10T01:41:41.408 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:41.408 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror image_30 snap 2022-04-10T01:41:41.408 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:41.409 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:41.409 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_30 2022-04-10T01:41:41.409 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:41:41.409 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:41:41.409 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:41.410 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:41.410 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:41.410 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:41.410 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:41:41.410 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:41:42.408 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/image_30@snap 2022-04-10T01:41:42.450 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_30': 2022-04-10T01:41:42.451 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:41:42.451 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:41:42.451 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 2 2022-04-10T01:41:42.451 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 113354f510dd 2022-04-10T01:41:42.452 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.113354f510dd 2022-04-10T01:41:42.452 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:41:42.452 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:41:42.452 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:41:42.452 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:41:42.453 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:33:22 2022 2022-04-10T01:41:42.453 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:33:22 2022 2022-04-10T01:41:42.453 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:33:22 2022 2022-04-10T01:41:42.453 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:41:42.454 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:41:42.454 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:41:42.454 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: e756a8db-50ef-4000-ba87-3241379f1d82 2022-04-10T01:41:42.454 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:41:42.456 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:42.456 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror image_30 snap 2022-04-10T01:41:42.456 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:42.456 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_30 2022-04-10T01:41:42.457 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:41:42.457 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:41:42.457 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_30.export 2022-04-10T01:41:42.457 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_30.export 2022-04-10T01:41:42.458 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_30.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_30.export 2022-04-10T01:41:42.458 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export image_30@snap /tmp/tmp.rgwz74WspX/cluster2-mirror-image_30.export 2022-04-10T01:41:42.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:41:42.648 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export image_30@snap /tmp/tmp.rgwz74WspX/cluster1-mirror-image_30.export 2022-04-10T01:41:42.885 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:41:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-image_30.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_30.export 2022-04-10T01:41:42.953 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_30.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_30.export 2022-04-10T01:41:42.992 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:42.993 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:41:42.993 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_31 2022-04-10T01:41:42.993 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror image_31 snap 2022-04-10T01:41:42.994 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:42.994 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:42.994 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_31 2022-04-10T01:41:42.994 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:41:42.994 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create image_31@snap --debug-rbd=20 --debug-journaler=20 2022-04-10T01:41:44.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror image_31 2022-04-10T01:41:44.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:44.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:44.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_31 2022-04-10T01:41:44.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:44.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:44.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:44.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:44.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror image_31 started 2022-04-10T01:41:44.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:44.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:44.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_31 2022-04-10T01:41:44.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:41:44.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:41:44.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:44.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:44.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:44.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:44.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:41:44.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:41:45.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror image_31 started 2022-04-10T01:41:45.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:45.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:45.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_31 2022-04-10T01:41:45.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:41:45.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:41:45.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:41:45.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:45.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:45.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:45.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:45.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_31 2022-04-10T01:41:45.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:41:45.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:41:45.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:41:45.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:41:45.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:41:45.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:41:45.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:41:45.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:41:45.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:45.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:41:45.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:41:45.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:41:45.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:41:45.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:41:45.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:41:45.532 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_31 2022-04-10T01:41:45.532 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:41:45.532 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:41:45.532 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:41:45.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:41:45.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:41:45.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:41:45.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:41:45.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:41:45.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:41:45.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:41:45.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:41:45.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:41:45.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:41:45.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:41:45.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:41:45.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:41:45.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:45.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:41:45.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:41:45.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:41:45.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:41:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:41:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:41:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:41:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:41:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:41:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:41:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:41:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:41:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:41:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:41:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:41:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:41:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:41:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:41:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:41:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:41:45.544 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:41:45.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:41:45.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:41:45.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_31 2022-04-10T01:41:45.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:41:45.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:41:45.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:41:45.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:41:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:41:45.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:41:45.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:41:45.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:45.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_31 2022-04-10T01:41:45.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:41:45.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_31 2022-04-10T01:41:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:41:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:45.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:45.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:41:45.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:41:45.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_31 2022-04-10T01:41:45.677 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:41:45.677 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:41:45.677 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:45.677 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_31 2022-04-10T01:41:45.677 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:45.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:45.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:45.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:45.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:41:45.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:45.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:45.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:45.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-image_31.status 2022-04-10T01:41:45.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-image_31.status 2022-04-10T01:41:45.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:41:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_31.status 2022-04-10T01:41:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-image_31.status 2022-04-10T01:41:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-image_31.status 2022-04-10T01:41:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:41:45.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_31.status 2022-04-10T01:41:45.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror image_31 2022-04-10T01:41:45.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:45.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:45.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_31 2022-04-10T01:41:45.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:41:45.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:45.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:45.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:45.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_31 2022-04-10T01:41:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 209 2022-04-10T01:41:46.621 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_31 /tmp/tmp.rgwz74WspX/cluster2-mirror-image_31.status 2022-04-10T01:41:46.621 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:46.621 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:46.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_31 2022-04-10T01:41:46.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_31.status 2022-04-10T01:41:46.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:41:46.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:46.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:46.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:46.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_31 --format xml 2022-04-10T01:41:46.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:41:46.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:41:46.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=209 2022-04-10T01:41:46.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:41:46.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:41:46.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:41:46.870 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_31.status 2022-04-10T01:41:46.870 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:46.870 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:46.870 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_31 2022-04-10T01:41:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_31.status 2022-04-10T01:41:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2022-04-10T01:41:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:41:46.920 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:41:46.921 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:41:46.923 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:41:46.923 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:41:46.923 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:41:46.924 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:41:46.924 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 209 = '' 2022-04-10T01:41:46.924 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:41:46.924 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:41:47.325 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_31.status 2022-04-10T01:41:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_31 2022-04-10T01:41:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_31.status 2022-04-10T01:41:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:47.327 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:47.327 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:47.327 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:47.327 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2022-04-10T01:41:47.327 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:41:47.372 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:41:47.373 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:41:47.374 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:41:47.374 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:41:47.374 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:41:47.375 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:41:47.376 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 209 = '' 2022-04-10T01:41:47.376 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:41:47.376 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:41:48.177 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_31.status 2022-04-10T01:41:48.178 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:48.178 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:48.178 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_31 2022-04-10T01:41:48.179 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_31.status 2022-04-10T01:41:48.179 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:48.180 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:48.180 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:48.180 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:48.181 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2022-04-10T01:41:48.181 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:41:48.220 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:41:48.223 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=209 2022-04-10T01:41:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 209 = 209 2022-04-10T01:41:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror image_31 snap 2022-04-10T01:41:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_31 2022-04-10T01:41:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:41:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:41:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:48.226 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:48.226 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:48.226 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:48.226 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:41:48.226 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:41:49.224 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/image_31@snap 2022-04-10T01:41:49.265 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_31': 2022-04-10T01:41:49.266 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:41:49.266 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:41:49.266 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 2 2022-04-10T01:41:49.266 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 118d67c03a5a 2022-04-10T01:41:49.267 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.118d67c03a5a 2022-04-10T01:41:49.267 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:41:49.267 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:41:49.267 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:41:49.267 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:41:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:33:36 2022 2022-04-10T01:41:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:33:36 2022 2022-04-10T01:41:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:33:36 2022 2022-04-10T01:41:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:41:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:41:49.269 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:41:49.269 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 9c9c599b-a61d-4229-a0b1-ba391a8116f1 2022-04-10T01:41:49.269 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:41:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:49.271 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror image_31 snap 2022-04-10T01:41:49.271 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:49.271 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_31 2022-04-10T01:41:49.271 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:41:49.271 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:41:49.272 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_31.export 2022-04-10T01:41:49.272 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_31.export 2022-04-10T01:41:49.272 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_31.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_31.export 2022-04-10T01:41:49.272 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export image_31@snap /tmp/tmp.rgwz74WspX/cluster2-mirror-image_31.export 2022-04-10T01:41:49.454 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:41:49.458 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export image_31@snap /tmp/tmp.rgwz74WspX/cluster1-mirror-image_31.export 2022-04-10T01:41:49.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:41:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-image_31.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_31.export 2022-04-10T01:41:49.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_31.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_31.export 2022-04-10T01:41:49.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:49.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:41:49.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_32 2022-04-10T01:41:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror image_32 snap 2022-04-10T01:41:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_32 2022-04-10T01:41:49.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:41:49.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create image_32@snap --debug-rbd=20 --debug-journaler=20 2022-04-10T01:41:50.869 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3048757474569679) > weight (0.25) 2022-04-10T01:41:50.869 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7759002659204196) > weight (0.25) 2022-04-10T01:41:50.869 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.40172347322492297) > weight (0.25) 2022-04-10T01:41:50.870 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.4528548212614595) > weight (0.25) 2022-04-10T01:41:50.870 INFO:tasks.rbd_mirror_thrash:waiting for 106 secs before thrashing 2022-04-10T01:41:51.630 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror image_32 2022-04-10T01:41:51.631 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:51.631 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:51.631 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_32 2022-04-10T01:41:51.632 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:51.632 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:51.632 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:51.632 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:51.632 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror image_32 started 2022-04-10T01:41:51.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:51.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:51.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_32 2022-04-10T01:41:51.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:41:51.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:41:51.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:51.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:51.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:51.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:51.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:41:51.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:41:52.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror image_32 started 2022-04-10T01:41:52.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:52.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_32 2022-04-10T01:41:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:41:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:41:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:41:52.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:52.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:52.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:52.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:52.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_32 2022-04-10T01:41:52.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:41:52.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:41:52.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:41:52.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:41:52.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:41:52.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:41:52.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:41:52.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:41:52.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:52.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:41:52.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:41:52.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:41:52.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:41:52.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:41:52.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:41:52.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_32 2022-04-10T01:41:52.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:41:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:41:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:41:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:41:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:41:52.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:41:52.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:41:52.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:41:52.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:41:52.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:41:52.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:41:52.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:41:52.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:41:52.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:41:52.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:41:52.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:41:52.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:52.647 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:41:52.647 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:41:52.647 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:41:52.647 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:41:52.648 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:41:52.648 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:41:52.648 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:41:52.648 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:41:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:41:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:41:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:41:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:41:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:41:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:41:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:41:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:41:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:41:52.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:41:52.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:52.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:41:52.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:41:52.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:41:52.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:41:52.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:41:52.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_32 2022-04-10T01:41:52.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2022-04-10T01:41:52.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:41:52.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:41:52.773 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:41:52.773 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_32 2022-04-10T01:41:52.773 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:1 2022-04-10T01:41:52.774 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:41:52.774 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:41:52.774 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:1 cluster instance 2022-04-10T01:41:52.774 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:1 2022-04-10T01:41:52.775 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:41:52.775 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:41:52.775 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:41:52.775 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=1 2022-04-10T01:41:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 1 = cluster1:1 ']' 2022-04-10T01:41:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:41:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:41:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=1 2022-04-10T01:41:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:41:52.777 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:41:52.777 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:41:52.777 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:52.777 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:41:52.778 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:1 cluster1 2022-04-10T01:41:52.778 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:1 2022-04-10T01:41:52.778 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:41:52.778 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:41:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2022-04-10T01:41:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:1 2022-04-10T01:41:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:41:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:41:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:41:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:41:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 1 = cluster1:1 ']' 2022-04-10T01:41:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:41:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:41:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=1 2022-04-10T01:41:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=1 2022-04-10T01:41:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:41:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:41:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:41:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2022-04-10T01:41:52.788 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:41:52.788 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:41:52.788 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:41:52.788 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_32 2022-04-10T01:41:52.895 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2022-04-10T01:41:52.896 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:41:52.896 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:41:52.900 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:41:52.900 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_32 2022-04-10T01:41:52.900 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:2 2022-04-10T01:41:52.901 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:41:52.901 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:41:52.901 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:2 cluster instance 2022-04-10T01:41:52.901 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:2 2022-04-10T01:41:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:41:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:41:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:41:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=2 2022-04-10T01:41:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 2 = cluster1:2 ']' 2022-04-10T01:41:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:41:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:41:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=2 2022-04-10T01:41:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:41:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:41:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:41:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:41:52.905 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:2 cluster1 2022-04-10T01:41:52.905 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:2 2022-04-10T01:41:52.905 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:41:52.905 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:41:52.905 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2022-04-10T01:41:52.906 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:2 2022-04-10T01:41:52.906 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:41:52.906 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:41:52.906 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:41:52.907 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:41:52.907 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 2 = cluster1:2 ']' 2022-04-10T01:41:52.907 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:41:52.907 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:41:52.907 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=2 2022-04-10T01:41:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=2 2022-04-10T01:41:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:41:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:41:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:52.909 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:41:52.909 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2022-04-10T01:41:52.915 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:41:52.915 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:41:52.915 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:41:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_32 2022-04-10T01:41:53.024 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2022-04-10T01:41:53.024 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:41:53.024 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:41:53.028 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:41:53.028 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_32 2022-04-10T01:41:53.029 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:3 2022-04-10T01:41:53.029 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:41:53.029 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:41:53.029 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:3 cluster instance 2022-04-10T01:41:53.030 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:3 2022-04-10T01:41:53.030 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:41:53.030 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:41:53.030 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:41:53.030 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=3 2022-04-10T01:41:53.031 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 3 = cluster1:3 ']' 2022-04-10T01:41:53.031 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:41:53.031 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:41:53.031 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=3 2022-04-10T01:41:53.032 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=3 2022-04-10T01:41:53.032 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:41:53.032 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:41:53.032 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:53.033 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:41:53.033 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:3 cluster1 2022-04-10T01:41:53.033 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:3 2022-04-10T01:41:53.033 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:41:53.034 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:41:53.034 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2022-04-10T01:41:53.034 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:3 2022-04-10T01:41:53.034 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:41:53.034 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:41:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:41:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=3 2022-04-10T01:41:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 3 = cluster1:3 ']' 2022-04-10T01:41:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:41:53.036 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:41:53.036 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=3 2022-04-10T01:41:53.036 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=3 2022-04-10T01:41:53.036 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:41:53.037 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:41:53.037 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:53.037 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:41:53.037 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2022-04-10T01:41:53.042 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:41:53.043 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:41:53.043 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:41:53.043 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_32 2022-04-10T01:41:53.157 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:41:53.158 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:41:53.158 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:41:53.158 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:41:53.159 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:41:53.163 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:41:53.163 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:41:53.164 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:53.164 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_32 2022-04-10T01:41:53.164 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:41:53.164 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:53.164 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:53.165 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_32 2022-04-10T01:41:53.165 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:41:53.165 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:53.165 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:53.165 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:53.166 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:41:53.166 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:41:53.166 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_32 2022-04-10T01:41:53.166 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:41:53.167 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:41:53.167 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:53.167 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_32 2022-04-10T01:41:53.167 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:53.167 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:53.168 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:53.168 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:53.168 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:41:53.168 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:53.168 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-image_32.status 2022-04-10T01:41:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-image_32.status 2022-04-10T01:41:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:41:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_32.status 2022-04-10T01:41:53.170 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-image_32.status 2022-04-10T01:41:53.170 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-image_32.status 2022-04-10T01:41:53.170 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:41:53.170 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_32.status 2022-04-10T01:41:53.170 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror image_32 2022-04-10T01:41:53.171 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:53.171 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:53.171 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_32 2022-04-10T01:41:53.171 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:41:53.171 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:53.172 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:53.172 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:53.172 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_32 2022-04-10T01:41:53.625 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 212 2022-04-10T01:41:53.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_32 /tmp/tmp.rgwz74WspX/cluster2-mirror-image_32.status 2022-04-10T01:41:53.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_32 2022-04-10T01:41:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_32.status 2022-04-10T01:41:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:41:53.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:53.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:53.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:53.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_32 --format xml 2022-04-10T01:41:53.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:41:53.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:41:53.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=212 2022-04-10T01:41:53.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:41:53.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:41:53.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:41:53.879 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_32.status 2022-04-10T01:41:53.880 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:53.880 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:53.880 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_32 2022-04-10T01:41:53.880 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_32.status 2022-04-10T01:41:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2022-04-10T01:41:53.882 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:41:53.925 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:41:53.925 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:41:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:41:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:41:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:41:53.928 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:41:53.928 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 212 = '' 2022-04-10T01:41:53.928 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:41:53.928 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:41:54.329 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_32.status 2022-04-10T01:41:54.329 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:54.330 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:54.330 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_32 2022-04-10T01:41:54.330 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_32.status 2022-04-10T01:41:54.330 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:54.331 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:54.331 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:54.331 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:54.331 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2022-04-10T01:41:54.332 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:41:54.378 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:41:54.378 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:41:54.380 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:41:54.380 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:41:54.380 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:41:54.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:41:54.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 212 = '' 2022-04-10T01:41:54.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:41:54.382 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:41:55.184 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_32.status 2022-04-10T01:41:55.184 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:55.185 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:55.185 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_32 2022-04-10T01:41:55.186 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_32.status 2022-04-10T01:41:55.186 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:55.187 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:55.187 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:55.187 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:55.188 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2022-04-10T01:41:55.188 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:41:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:41:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=212 2022-04-10T01:41:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 212 = 212 2022-04-10T01:41:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror image_32 snap 2022-04-10T01:41:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_32 2022-04-10T01:41:55.233 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:41:55.233 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:41:55.233 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:55.234 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:55.234 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:55.235 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:55.235 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:41:55.235 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:41:56.228 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/image_32@snap 2022-04-10T01:41:56.270 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_32': 2022-04-10T01:41:56.271 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:41:56.271 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:41:56.271 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 2 2022-04-10T01:41:56.272 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 12a7dd04c2dc 2022-04-10T01:41:56.272 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.12a7dd04c2dc 2022-04-10T01:41:56.272 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:41:56.272 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:41:56.272 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:41:56.273 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:41:56.273 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:33:47 2022 2022-04-10T01:41:56.273 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:33:47 2022 2022-04-10T01:41:56.273 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:33:47 2022 2022-04-10T01:41:56.274 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:41:56.274 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:41:56.274 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:41:56.274 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 34f68cc3-808d-4b7f-a2e3-415ae64356a9 2022-04-10T01:41:56.275 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:41:56.276 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:56.276 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror image_32 snap 2022-04-10T01:41:56.276 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:56.277 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_32 2022-04-10T01:41:56.277 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:41:56.277 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:41:56.277 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_32.export 2022-04-10T01:41:56.278 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_32.export 2022-04-10T01:41:56.278 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_32.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_32.export 2022-04-10T01:41:56.278 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export image_32@snap /tmp/tmp.rgwz74WspX/cluster2-mirror-image_32.export 2022-04-10T01:41:56.455 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:41:56.458 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export image_32@snap /tmp/tmp.rgwz74WspX/cluster1-mirror-image_32.export 2022-04-10T01:41:56.704 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:41:56.707 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-image_32.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_32.export 2022-04-10T01:41:56.771 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_32.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_32.export 2022-04-10T01:41:56.809 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:56.809 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:41:56.810 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_33 2022-04-10T01:41:56.810 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror image_33 snap 2022-04-10T01:41:56.810 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:56.810 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:56.811 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_33 2022-04-10T01:41:56.811 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:41:56.811 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create image_33@snap --debug-rbd=20 --debug-journaler=20 2022-04-10T01:41:58.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror image_33 2022-04-10T01:41:58.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:58.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:58.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_33 2022-04-10T01:41:58.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:58.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:58.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:58.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:58.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror image_33 started 2022-04-10T01:41:58.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:58.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:58.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_33 2022-04-10T01:41:58.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:41:58.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:41:58.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:58.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:58.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:58.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:58.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:41:58.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:41:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror image_33 started 2022-04-10T01:41:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_33 2022-04-10T01:41:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:41:59.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:41:59.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:41:59.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:59.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:59.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:59.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:59.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_33 2022-04-10T01:41:59.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:41:59.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:41:59.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:41:59.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:41:59.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:41:59.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:41:59.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:41:59.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:41:59.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:59.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:41:59.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:41:59.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:41:59.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:41:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:41:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:41:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2022-04-10T01:41:59.647 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:41:59.647 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:41:59.647 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:41:59.647 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:41:59.648 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:41:59.648 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:41:59.648 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:41:59.648 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:41:59.649 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:41:59.649 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:41:59.649 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:41:59.649 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:41:59.649 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:41:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:41:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:41:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:41:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:59.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:41:59.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:41:59.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:41:59.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:41:59.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:41:59.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:41:59.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:41:59.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:41:59.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:41:59.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:41:59.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:41:59.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:41:59.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:41:59.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:41:59.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:41:59.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:41:59.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:41:59.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:41:59.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:59.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:41:59.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:41:59.658 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:41:59.659 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:41:59.659 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:41:59.659 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2022-04-10T01:41:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2022-04-10T01:41:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:41:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:41:59.776 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:41:59.776 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_33 2022-04-10T01:41:59.776 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:1 2022-04-10T01:41:59.777 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:41:59.777 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:41:59.777 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:1 cluster instance 2022-04-10T01:41:59.777 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:1 2022-04-10T01:41:59.777 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:41:59.778 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:41:59.778 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:41:59.778 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=1 2022-04-10T01:41:59.778 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 1 = cluster1:1 ']' 2022-04-10T01:41:59.779 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:41:59.779 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:41:59.779 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=1 2022-04-10T01:41:59.779 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:41:59.779 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:41:59.780 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:41:59.780 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:59.780 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:41:59.780 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:1 cluster1 2022-04-10T01:41:59.780 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:1 2022-04-10T01:41:59.781 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:41:59.781 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:41:59.781 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2022-04-10T01:41:59.781 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:1 2022-04-10T01:41:59.782 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:41:59.782 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:41:59.782 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:41:59.782 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:41:59.782 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 1 = cluster1:1 ']' 2022-04-10T01:41:59.783 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:41:59.783 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:41:59.783 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=1 2022-04-10T01:41:59.783 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=1 2022-04-10T01:41:59.784 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:41:59.784 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:41:59.784 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:59.784 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:41:59.784 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2022-04-10T01:41:59.790 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:41:59.791 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:41:59.791 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:41:59.791 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_33 2022-04-10T01:41:59.904 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:41:59.904 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:41:59.905 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:41:59.905 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:41:59.905 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:41:59.908 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:41:59.908 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:41:59.909 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:59.909 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_33 2022-04-10T01:41:59.909 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:41:59.909 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:59.909 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:59.910 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_33 2022-04-10T01:41:59.910 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:41:59.910 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:59.910 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:59.911 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:59.911 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:41:59.911 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:41:59.911 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_33 2022-04-10T01:41:59.911 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:41:59.911 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:41:59.912 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:59.912 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_33 2022-04-10T01:41:59.912 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:41:59.912 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:41:59.912 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:59.913 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:59.913 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:41:59.913 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:59.913 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:59.913 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:59.914 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-image_33.status 2022-04-10T01:41:59.914 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-image_33.status 2022-04-10T01:41:59.914 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:41:59.914 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_33.status 2022-04-10T01:41:59.914 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-image_33.status 2022-04-10T01:41:59.915 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-image_33.status 2022-04-10T01:41:59.915 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:41:59.916 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_33.status 2022-04-10T01:41:59.916 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror image_33 2022-04-10T01:41:59.917 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:59.917 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:41:59.917 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_33 2022-04-10T01:41:59.917 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:41:59.917 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:41:59.918 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:41:59.918 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:41:59.918 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_33 2022-04-10T01:42:00.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 215 2022-04-10T01:42:00.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_33 /tmp/tmp.rgwz74WspX/cluster2-mirror-image_33.status 2022-04-10T01:42:00.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:00.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:00.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_33 2022-04-10T01:42:00.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_33.status 2022-04-10T01:42:00.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:42:00.658 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:00.658 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:00.659 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:00.659 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_33 --format xml 2022-04-10T01:42:00.659 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:42:00.698 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:42:00.700 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=215 2022-04-10T01:42:00.700 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:42:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:42:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:42:00.902 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_33.status 2022-04-10T01:42:00.902 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:00.903 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:00.903 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_33 2022-04-10T01:42:00.903 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_33.status 2022-04-10T01:42:00.903 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:00.903 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2022-04-10T01:42:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:42:00.947 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:42:00.948 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:42:00.949 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:42:00.949 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:42:00.949 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:42:00.950 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:42:00.950 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 215 = '' 2022-04-10T01:42:00.951 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:42:00.951 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:42:01.352 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_33.status 2022-04-10T01:42:01.352 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:01.352 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:01.352 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_33 2022-04-10T01:42:01.353 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_33.status 2022-04-10T01:42:01.353 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:01.353 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:01.353 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:01.353 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:01.354 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2022-04-10T01:42:01.354 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:42:01.399 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:42:01.399 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:42:01.401 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:42:01.401 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:42:01.401 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:42:01.402 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:42:01.402 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 215 = '' 2022-04-10T01:42:01.402 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:42:01.403 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:42:02.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_33.status 2022-04-10T01:42:02.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:02.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:02.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_33 2022-04-10T01:42:02.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_33.status 2022-04-10T01:42:02.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:02.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:02.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:02.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:02.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2022-04-10T01:42:02.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:42:02.244 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:42:02.247 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=215 2022-04-10T01:42:02.247 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 215 = 215 2022-04-10T01:42:02.248 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:02.248 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror image_33 snap 2022-04-10T01:42:02.248 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:02.248 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:02.248 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_33 2022-04-10T01:42:02.249 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:42:02.249 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:42:02.249 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:02.249 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:02.250 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:02.250 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:02.250 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:42:02.250 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:42:03.248 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/image_33@snap 2022-04-10T01:42:03.291 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_33': 2022-04-10T01:42:03.291 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:42:03.291 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:42:03.292 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 2 2022-04-10T01:42:03.292 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 10e4df88ba2f 2022-04-10T01:42:03.292 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.10e4df88ba2f 2022-04-10T01:42:03.292 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:42:03.292 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:42:03.293 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:42:03.293 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:42:03.293 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:34:02 2022 2022-04-10T01:42:03.293 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:34:02 2022 2022-04-10T01:42:03.293 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:34:02 2022 2022-04-10T01:42:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:42:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:42:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:42:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 3334b7aa-6174-45c6-b4b8-608b7f09db2a 2022-04-10T01:42:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:42:03.296 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:03.296 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror image_33 snap 2022-04-10T01:42:03.296 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:03.296 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_33 2022-04-10T01:42:03.297 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:42:03.297 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:42:03.297 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_33.export 2022-04-10T01:42:03.297 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_33.export 2022-04-10T01:42:03.297 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_33.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_33.export 2022-04-10T01:42:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export image_33@snap /tmp/tmp.rgwz74WspX/cluster2-mirror-image_33.export 2022-04-10T01:42:03.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:42:03.483 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export image_33@snap /tmp/tmp.rgwz74WspX/cluster1-mirror-image_33.export 2022-04-10T01:42:03.723 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:42:03.726 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-image_33.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_33.export 2022-04-10T01:42:03.790 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_33.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_33.export 2022-04-10T01:42:03.827 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:03.828 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:42:03.828 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_34 2022-04-10T01:42:03.828 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror image_34 snap 2022-04-10T01:42:03.828 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:03.829 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:03.829 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_34 2022-04-10T01:42:03.829 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:42:03.829 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create image_34@snap --debug-rbd=20 --debug-journaler=20 2022-04-10T01:42:05.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror image_34 2022-04-10T01:42:05.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:05.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:05.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_34 2022-04-10T01:42:05.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror image_34 started 2022-04-10T01:42:05.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:05.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:05.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_34 2022-04-10T01:42:05.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:42:05.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:42:05.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:05.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:05.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:05.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:05.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:42:05.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:42:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror image_34 started 2022-04-10T01:42:06.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:06.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:06.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_34 2022-04-10T01:42:06.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:42:06.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:42:06.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:42:06.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:06.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:06.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:06.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:06.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_34 2022-04-10T01:42:06.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:42:06.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:42:06.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:42:06.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:42:06.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:42:06.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:42:06.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:42:06.528 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:42:06.528 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:06.528 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:42:06.528 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:42:06.529 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:42:06.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:42:06.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:42:06.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:42:06.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_34 2022-04-10T01:42:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:42:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:42:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:42:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:42:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:42:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:42:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:42:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:42:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:42:06.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:42:06.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:42:06.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:42:06.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:42:06.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:42:06.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:42:06.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:42:06.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:06.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:42:06.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:42:06.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:42:06.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:42:06.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:42:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:42:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:42:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:42:06.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:42:06.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:42:06.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:42:06.541 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:42:06.541 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:42:06.541 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:42:06.541 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:42:06.541 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:42:06.542 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:42:06.542 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:42:06.542 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:06.542 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:42:06.543 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:42:06.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:42:06.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:42:06.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:42:06.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_34 2022-04-10T01:42:06.661 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2022-04-10T01:42:06.661 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:42:06.661 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:42:06.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:42:06.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_34 2022-04-10T01:42:06.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:1 2022-04-10T01:42:06.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:42:06.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:42:06.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:1 cluster instance 2022-04-10T01:42:06.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:1 2022-04-10T01:42:06.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:42:06.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:42:06.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:42:06.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=1 2022-04-10T01:42:06.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 1 = cluster1:1 ']' 2022-04-10T01:42:06.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:42:06.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:42:06.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=1 2022-04-10T01:42:06.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:42:06.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:42:06.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:42:06.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:06.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:42:06.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:1 cluster1 2022-04-10T01:42:06.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:1 2022-04-10T01:42:06.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:42:06.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:42:06.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2022-04-10T01:42:06.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:1 2022-04-10T01:42:06.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:42:06.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:42:06.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:42:06.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:42:06.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 1 = cluster1:1 ']' 2022-04-10T01:42:06.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:42:06.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:42:06.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=1 2022-04-10T01:42:06.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=1 2022-04-10T01:42:06.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:42:06.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:42:06.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:06.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:42:06.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2022-04-10T01:42:06.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:42:06.681 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:42:06.681 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:42:06.681 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_34 2022-04-10T01:42:06.794 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:42:06.795 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:42:06.795 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:42:06.795 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:42:06.796 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:42:06.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:42:06.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:42:06.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:06.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_34 2022-04-10T01:42:06.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:42:06.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:06.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:06.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_34 2022-04-10T01:42:06.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:42:06.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:06.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:06.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:06.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:42:06.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:42:06.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_34 2022-04-10T01:42:06.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:42:06.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:42:06.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:06.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_34 2022-04-10T01:42:06.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:06.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:06.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:06.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:06.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:42:06.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:06.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:06.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:06.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-image_34.status 2022-04-10T01:42:06.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-image_34.status 2022-04-10T01:42:06.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:42:06.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_34.status 2022-04-10T01:42:06.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-image_34.status 2022-04-10T01:42:06.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-image_34.status 2022-04-10T01:42:06.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:42:06.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_34.status 2022-04-10T01:42:06.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror image_34 2022-04-10T01:42:06.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:06.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:06.809 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_34 2022-04-10T01:42:06.809 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:42:06.809 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:06.809 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:06.810 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:06.810 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_34 2022-04-10T01:42:07.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 218 2022-04-10T01:42:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_34 /tmp/tmp.rgwz74WspX/cluster2-mirror-image_34.status 2022-04-10T01:42:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_34 2022-04-10T01:42:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_34.status 2022-04-10T01:42:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:42:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_34 --format xml 2022-04-10T01:42:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:42:07.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:42:07.583 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=218 2022-04-10T01:42:07.583 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:42:07.583 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:42:07.584 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:42:07.785 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_34.status 2022-04-10T01:42:07.785 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:07.786 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:07.786 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_34 2022-04-10T01:42:07.786 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_34.status 2022-04-10T01:42:07.786 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2022-04-10T01:42:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:42:07.831 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:42:07.831 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:42:07.833 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:42:07.833 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:42:07.834 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:42:07.834 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:42:07.834 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 218 = '' 2022-04-10T01:42:07.834 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:42:07.834 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:42:08.235 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_34.status 2022-04-10T01:42:08.235 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:08.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:08.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_34 2022-04-10T01:42:08.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_34.status 2022-04-10T01:42:08.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:08.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:08.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:08.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:08.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2022-04-10T01:42:08.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:42:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:42:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:42:08.290 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:42:08.290 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:42:08.291 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:42:08.291 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:42:08.291 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 218 = '' 2022-04-10T01:42:08.291 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:42:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:42:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_34.status 2022-04-10T01:42:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_34 2022-04-10T01:42:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_34.status 2022-04-10T01:42:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:09.095 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2022-04-10T01:42:09.095 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:42:09.134 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:42:09.137 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=218 2022-04-10T01:42:09.137 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 218 = 218 2022-04-10T01:42:09.137 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:09.138 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror image_34 snap 2022-04-10T01:42:09.138 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:09.138 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:09.138 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_34 2022-04-10T01:42:09.139 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:42:09.139 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:42:09.139 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:09.139 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:09.139 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:09.140 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:09.140 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:42:09.140 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:42:10.137 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/image_34@snap 2022-04-10T01:42:10.179 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_34': 2022-04-10T01:42:10.179 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:42:10.179 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:42:10.179 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 2 2022-04-10T01:42:10.180 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 1133a0980c8c 2022-04-10T01:42:10.180 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.1133a0980c8c 2022-04-10T01:42:10.180 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:42:10.180 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:42:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:42:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:42:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:34:17 2022 2022-04-10T01:42:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:34:17 2022 2022-04-10T01:42:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:34:17 2022 2022-04-10T01:42:10.182 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:42:10.182 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:42:10.182 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:42:10.182 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: fc8271db-fe00-47f6-a840-9d97ed18fbb7 2022-04-10T01:42:10.183 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:42:10.184 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:10.184 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror image_34 snap 2022-04-10T01:42:10.184 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_34 2022-04-10T01:42:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:42:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:42:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_34.export 2022-04-10T01:42:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_34.export 2022-04-10T01:42:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_34.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_34.export 2022-04-10T01:42:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export image_34@snap /tmp/tmp.rgwz74WspX/cluster2-mirror-image_34.export 2022-04-10T01:42:10.374 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:42:10.377 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export image_34@snap /tmp/tmp.rgwz74WspX/cluster1-mirror-image_34.export 2022-04-10T01:42:10.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:42:10.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-image_34.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_34.export 2022-04-10T01:42:10.647 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_34.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_34.export 2022-04-10T01:42:10.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:10.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:42:10.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_35 2022-04-10T01:42:10.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror image_35 snap 2022-04-10T01:42:10.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:10.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:10.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_35 2022-04-10T01:42:10.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:42:10.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create image_35@snap --debug-rbd=20 --debug-journaler=20 2022-04-10T01:42:12.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror image_35 2022-04-10T01:42:12.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:12.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:12.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_35 2022-04-10T01:42:12.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:12.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:12.541 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:12.541 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:12.541 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror image_35 started 2022-04-10T01:42:12.541 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:12.541 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:12.542 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_35 2022-04-10T01:42:12.542 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:42:12.542 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:42:12.542 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:12.543 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:12.543 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:12.543 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:12.543 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:42:12.543 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:42:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror image_35 started 2022-04-10T01:42:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_35 2022-04-10T01:42:13.543 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:42:13.543 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:42:13.543 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:42:13.543 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:13.544 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:13.544 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:13.544 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:13.544 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_35 2022-04-10T01:42:13.544 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:42:13.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:42:13.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:42:13.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:42:13.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:42:13.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:42:13.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:42:13.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:42:13.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:13.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:42:13.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:42:13.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:42:13.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:42:13.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:42:13.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:42:13.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_35 2022-04-10T01:42:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:42:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:42:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:42:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:42:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:42:13.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:42:13.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:42:13.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:42:13.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:42:13.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:42:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:42:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:42:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:42:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:42:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:42:13.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:42:13.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:13.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:42:13.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:42:13.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:42:13.556 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:42:13.556 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:42:13.556 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:42:13.556 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:42:13.556 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:42:13.557 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:42:13.557 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:42:13.557 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:42:13.557 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:42:13.557 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:42:13.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:42:13.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:42:13.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:42:13.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:42:13.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:42:13.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:13.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:42:13.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:42:13.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:42:13.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:42:13.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:42:13.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_35 2022-04-10T01:42:13.677 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2022-04-10T01:42:13.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:42:13.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:42:13.681 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:42:13.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_35 2022-04-10T01:42:13.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:1 2022-04-10T01:42:13.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:42:13.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:42:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:1 cluster instance 2022-04-10T01:42:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:1 2022-04-10T01:42:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:42:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:42:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:42:13.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=1 2022-04-10T01:42:13.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 1 = cluster1:1 ']' 2022-04-10T01:42:13.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:42:13.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:42:13.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=1 2022-04-10T01:42:13.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:42:13.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:42:13.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:42:13.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:13.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:42:13.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:1 cluster1 2022-04-10T01:42:13.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:1 2022-04-10T01:42:13.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:42:13.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:42:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2022-04-10T01:42:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:1 2022-04-10T01:42:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:42:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:42:13.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:42:13.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:42:13.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 1 = cluster1:1 ']' 2022-04-10T01:42:13.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:42:13.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:42:13.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=1 2022-04-10T01:42:13.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=1 2022-04-10T01:42:13.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:42:13.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:42:13.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:13.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:42:13.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2022-04-10T01:42:13.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:42:13.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:42:13.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:42:13.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_35 2022-04-10T01:42:13.809 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:42:13.810 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:42:13.810 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:42:13.811 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:42:13.811 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:42:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:42:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:42:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_35 2022-04-10T01:42:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:42:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_35 2022-04-10T01:42:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:42:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:42:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:42:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_35 2022-04-10T01:42:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:42:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:42:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_35 2022-04-10T01:42:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:42:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-image_35.status 2022-04-10T01:42:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-image_35.status 2022-04-10T01:42:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:42:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_35.status 2022-04-10T01:42:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-image_35.status 2022-04-10T01:42:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-image_35.status 2022-04-10T01:42:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:42:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_35.status 2022-04-10T01:42:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror image_35 2022-04-10T01:42:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_35 2022-04-10T01:42:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:42:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_35 2022-04-10T01:42:14.543 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 221 2022-04-10T01:42:14.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_35 /tmp/tmp.rgwz74WspX/cluster2-mirror-image_35.status 2022-04-10T01:42:14.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:14.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:14.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_35 2022-04-10T01:42:14.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_35.status 2022-04-10T01:42:14.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:42:14.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:14.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:14.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:14.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_35 --format xml 2022-04-10T01:42:14.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:42:14.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:42:14.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=221 2022-04-10T01:42:14.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:42:14.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:42:14.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:42:14.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_35.status 2022-04-10T01:42:14.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:14.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:14.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_35 2022-04-10T01:42:14.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_35.status 2022-04-10T01:42:14.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:14.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:14.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:14.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:14.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2022-04-10T01:42:14.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:42:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:42:14.848 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:42:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:42:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:42:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:42:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:42:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 221 = '' 2022-04-10T01:42:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:42:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:42:15.252 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_35.status 2022-04-10T01:42:15.252 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_35 2022-04-10T01:42:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_35.status 2022-04-10T01:42:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:15.254 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:15.254 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:15.254 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2022-04-10T01:42:15.254 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:42:15.301 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:42:15.302 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:42:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:42:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:42:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:42:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:42:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 221 = '' 2022-04-10T01:42:15.305 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:42:15.305 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:42:16.106 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_35.status 2022-04-10T01:42:16.106 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:16.106 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:16.107 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_35 2022-04-10T01:42:16.107 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_35.status 2022-04-10T01:42:16.107 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:16.107 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:16.108 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:16.108 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:16.108 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2022-04-10T01:42:16.108 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:42:16.146 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:42:16.148 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=221 2022-04-10T01:42:16.149 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 221 = 221 2022-04-10T01:42:16.149 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:16.149 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror image_35 snap 2022-04-10T01:42:16.149 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_35 2022-04-10T01:42:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:42:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:42:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:16.151 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:16.151 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:16.151 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:16.151 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:42:16.152 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:42:17.149 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/image_35@snap 2022-04-10T01:42:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_35': 2022-04-10T01:42:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:42:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:42:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 2 2022-04-10T01:42:17.193 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 118d56d498e9 2022-04-10T01:42:17.193 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.118d56d498e9 2022-04-10T01:42:17.193 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:42:17.193 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:42:17.193 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:42:17.194 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:42:17.194 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:34:30 2022 2022-04-10T01:42:17.194 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:34:30 2022 2022-04-10T01:42:17.194 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:34:30 2022 2022-04-10T01:42:17.195 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:42:17.195 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:42:17.195 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:42:17.195 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 2864f9d0-8af5-4721-a606-e52b47ecb56e 2022-04-10T01:42:17.196 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:42:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror image_35 snap 2022-04-10T01:42:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_35 2022-04-10T01:42:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:42:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:42:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_35.export 2022-04-10T01:42:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_35.export 2022-04-10T01:42:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_35.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_35.export 2022-04-10T01:42:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export image_35@snap /tmp/tmp.rgwz74WspX/cluster2-mirror-image_35.export 2022-04-10T01:42:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:42:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export image_35@snap /tmp/tmp.rgwz74WspX/cluster1-mirror-image_35.export 2022-04-10T01:42:17.625 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:42:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-image_35.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_35.export 2022-04-10T01:42:17.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_35.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_35.export 2022-04-10T01:42:17.730 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:17.731 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:42:17.731 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_36 2022-04-10T01:42:17.731 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror image_36 snap 2022-04-10T01:42:17.732 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:17.732 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:17.732 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_36 2022-04-10T01:42:17.732 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:42:17.732 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create image_36@snap --debug-rbd=20 --debug-journaler=20 2022-04-10T01:42:19.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror image_36 2022-04-10T01:42:19.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:19.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:19.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_36 2022-04-10T01:42:19.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:19.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:19.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:19.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:19.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror image_36 started 2022-04-10T01:42:19.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:19.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:19.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_36 2022-04-10T01:42:19.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:42:19.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:42:19.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:19.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:19.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:19.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:19.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:42:19.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:42:20.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror image_36 started 2022-04-10T01:42:20.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:20.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:20.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_36 2022-04-10T01:42:20.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:42:20.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:42:20.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:42:20.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:20.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:20.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:20.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:20.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_36 2022-04-10T01:42:20.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:42:20.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:42:20.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:42:20.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:42:20.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:42:20.541 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:42:20.541 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:42:20.541 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:42:20.541 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:20.542 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:42:20.542 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:42:20.542 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:42:20.542 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:42:20.543 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:42:20.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:42:20.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_36 2022-04-10T01:42:20.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:42:20.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:42:20.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:42:20.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:42:20.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:42:20.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:42:20.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:42:20.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:42:20.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:42:20.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:42:20.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:42:20.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:42:20.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:42:20.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:42:20.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:42:20.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:42:20.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:20.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:42:20.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:42:20.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:42:20.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:42:20.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:42:20.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:42:20.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:42:20.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:42:20.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:42:20.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:42:20.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:42:20.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:42:20.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:42:20.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:42:20.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:42:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:42:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:42:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:42:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:42:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:42:20.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:42:20.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:42:20.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:42:20.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_36 2022-04-10T01:42:20.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2022-04-10T01:42:20.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:42:20.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:42:20.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:42:20.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_36 2022-04-10T01:42:20.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:1 2022-04-10T01:42:20.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:42:20.677 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:42:20.677 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:1 cluster instance 2022-04-10T01:42:20.677 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:1 2022-04-10T01:42:20.677 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:42:20.677 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:42:20.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:42:20.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=1 2022-04-10T01:42:20.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 1 = cluster1:1 ']' 2022-04-10T01:42:20.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:42:20.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:42:20.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=1 2022-04-10T01:42:20.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:42:20.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:42:20.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:42:20.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:20.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:42:20.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:1 cluster1 2022-04-10T01:42:20.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:1 2022-04-10T01:42:20.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:42:20.681 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:42:20.681 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2022-04-10T01:42:20.681 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:1 2022-04-10T01:42:20.681 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:42:20.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:42:20.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:42:20.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:42:20.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 1 = cluster1:1 ']' 2022-04-10T01:42:20.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:42:20.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:42:20.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=1 2022-04-10T01:42:20.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=1 2022-04-10T01:42:20.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:42:20.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:42:20.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:20.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:42:20.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2022-04-10T01:42:20.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:42:20.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:42:20.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:42:20.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_36 2022-04-10T01:42:20.795 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2022-04-10T01:42:20.795 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:42:20.796 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:42:20.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:42:20.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_36 2022-04-10T01:42:20.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:2 2022-04-10T01:42:20.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:42:20.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:42:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:2 cluster instance 2022-04-10T01:42:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:2 2022-04-10T01:42:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:42:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:42:20.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:42:20.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=2 2022-04-10T01:42:20.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 2 = cluster1:2 ']' 2022-04-10T01:42:20.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:42:20.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:42:20.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=2 2022-04-10T01:42:20.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:42:20.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:42:20.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:42:20.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:20.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:42:20.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:2 cluster1 2022-04-10T01:42:20.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:2 2022-04-10T01:42:20.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:42:20.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:42:20.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2022-04-10T01:42:20.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:2 2022-04-10T01:42:20.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:42:20.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:42:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:42:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:42:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 2 = cluster1:2 ']' 2022-04-10T01:42:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:42:20.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:42:20.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=2 2022-04-10T01:42:20.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=2 2022-04-10T01:42:20.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:42:20.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:42:20.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:20.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:42:20.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2022-04-10T01:42:20.813 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:42:20.813 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:42:20.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:42:20.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_36 2022-04-10T01:42:20.919 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2022-04-10T01:42:20.919 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:42:20.920 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:42:20.924 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:42:20.924 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_36 2022-04-10T01:42:20.924 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:3 2022-04-10T01:42:20.924 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:42:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:42:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:3 cluster instance 2022-04-10T01:42:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:3 2022-04-10T01:42:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:42:20.926 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:42:20.926 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:42:20.926 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=3 2022-04-10T01:42:20.926 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 3 = cluster1:3 ']' 2022-04-10T01:42:20.927 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:42:20.927 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:42:20.927 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=3 2022-04-10T01:42:20.927 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=3 2022-04-10T01:42:20.928 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:42:20.928 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:42:20.928 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:20.928 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:42:20.928 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:3 cluster1 2022-04-10T01:42:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:3 2022-04-10T01:42:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:42:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:42:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2022-04-10T01:42:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:3 2022-04-10T01:42:20.930 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:42:20.930 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:42:20.930 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:42:20.930 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=3 2022-04-10T01:42:20.930 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 3 = cluster1:3 ']' 2022-04-10T01:42:20.931 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:42:20.931 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:42:20.931 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=3 2022-04-10T01:42:20.931 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=3 2022-04-10T01:42:20.931 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:42:20.932 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:42:20.932 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:20.932 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:42:20.932 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2022-04-10T01:42:20.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:42:20.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:42:20.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:42:20.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_36 2022-04-10T01:42:21.053 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:42:21.054 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:42:21.054 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:42:21.054 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:42:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:42:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:42:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:42:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_36 2022-04-10T01:42:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:42:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_36 2022-04-10T01:42:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:42:21.060 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:21.060 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:21.060 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:21.060 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:42:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:42:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_36 2022-04-10T01:42:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:42:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:42:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_36 2022-04-10T01:42:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:21.063 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:42:21.063 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:21.063 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:21.063 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:21.063 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-image_36.status 2022-04-10T01:42:21.064 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-image_36.status 2022-04-10T01:42:21.064 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:42:21.064 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_36.status 2022-04-10T01:42:21.064 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-image_36.status 2022-04-10T01:42:21.064 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-image_36.status 2022-04-10T01:42:21.065 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:42:21.066 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_36.status 2022-04-10T01:42:21.066 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror image_36 2022-04-10T01:42:21.066 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:21.067 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:21.067 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_36 2022-04-10T01:42:21.067 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:42:21.067 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:21.067 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:21.068 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:21.068 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_36 2022-04-10T01:42:21.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 224 2022-04-10T01:42:21.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_36 /tmp/tmp.rgwz74WspX/cluster2-mirror-image_36.status 2022-04-10T01:42:21.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:21.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:21.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_36 2022-04-10T01:42:21.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_36.status 2022-04-10T01:42:21.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:42:21.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:21.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:21.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:21.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_36 --format xml 2022-04-10T01:42:21.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:42:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:42:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=224 2022-04-10T01:42:21.614 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:42:21.614 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:42:21.614 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:42:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_36.status 2022-04-10T01:42:21.816 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:21.816 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:21.816 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_36 2022-04-10T01:42:21.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_36.status 2022-04-10T01:42:21.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:21.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:21.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:21.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:21.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2022-04-10T01:42:21.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:42:21.859 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:42:21.860 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:42:21.860 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:42:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:42:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:42:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:42:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 224 = '' 2022-04-10T01:42:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:42:21.863 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:42:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_36.status 2022-04-10T01:42:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:22.264 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:22.264 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_36 2022-04-10T01:42:22.264 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_36.status 2022-04-10T01:42:22.264 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:22.265 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:22.265 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:22.265 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:22.265 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2022-04-10T01:42:22.265 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:42:22.310 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:42:22.310 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:42:22.312 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:42:22.313 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:42:22.313 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:42:22.313 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:42:22.313 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 224 = '' 2022-04-10T01:42:22.314 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:42:22.314 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:42:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_36.status 2022-04-10T01:42:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_36 2022-04-10T01:42:23.116 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_36.status 2022-04-10T01:42:23.116 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:23.116 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:23.116 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2022-04-10T01:42:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:42:23.154 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:42:23.158 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=224 2022-04-10T01:42:23.158 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 224 = 224 2022-04-10T01:42:23.159 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:23.159 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror image_36 snap 2022-04-10T01:42:23.159 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:23.160 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:23.160 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_36 2022-04-10T01:42:23.160 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:42:23.161 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:42:23.161 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:23.162 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:23.162 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:23.163 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:23.163 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:42:23.163 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:42:24.157 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/image_36@snap 2022-04-10T01:42:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_36': 2022-04-10T01:42:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:42:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:42:24.200 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 2 2022-04-10T01:42:24.200 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 12a7852cdd17 2022-04-10T01:42:24.200 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.12a7852cdd17 2022-04-10T01:42:24.200 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:42:24.201 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:42:24.201 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:42:24.201 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:42:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:34:42 2022 2022-04-10T01:42:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:34:42 2022 2022-04-10T01:42:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:34:42 2022 2022-04-10T01:42:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:42:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:42:24.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:42:24.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 347e3669-52d8-4e7d-9693-a1920cf47072 2022-04-10T01:42:24.203 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:42:24.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:24.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror image_36 snap 2022-04-10T01:42:24.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:24.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_36 2022-04-10T01:42:24.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:42:24.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:42:24.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_36.export 2022-04-10T01:42:24.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_36.export 2022-04-10T01:42:24.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_36.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_36.export 2022-04-10T01:42:24.207 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export image_36@snap /tmp/tmp.rgwz74WspX/cluster2-mirror-image_36.export 2022-04-10T01:42:24.395 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:42:24.399 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export image_36@snap /tmp/tmp.rgwz74WspX/cluster1-mirror-image_36.export 2022-04-10T01:42:24.852 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:42:24.855 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-image_36.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_36.export 2022-04-10T01:42:24.918 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_36.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_36.export 2022-04-10T01:42:24.955 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:24.956 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:42:24.956 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_37 2022-04-10T01:42:24.956 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror image_37 snap 2022-04-10T01:42:24.956 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:24.957 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:24.957 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_37 2022-04-10T01:42:24.957 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:42:24.957 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create image_37@snap --debug-rbd=20 --debug-journaler=20 2022-04-10T01:42:26.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror image_37 2022-04-10T01:42:26.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:26.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:26.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_37 2022-04-10T01:42:26.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:26.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:26.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:26.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:26.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror image_37 started 2022-04-10T01:42:26.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:26.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:26.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_37 2022-04-10T01:42:26.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:42:26.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:42:26.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:26.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:26.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:26.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:26.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:42:26.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:42:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror image_37 started 2022-04-10T01:42:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_37 2022-04-10T01:42:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:42:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:42:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:42:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:27.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:27.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_37 2022-04-10T01:42:27.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:42:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:42:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:42:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:42:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:42:27.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:42:27.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:42:27.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:42:27.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:27.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:42:27.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:42:27.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:42:27.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:42:27.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:42:27.588 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:42:27.588 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_37 2022-04-10T01:42:27.588 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:42:27.589 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:42:27.589 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:42:27.589 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:42:27.590 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:42:27.590 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:42:27.590 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:42:27.591 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:42:27.591 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:42:27.592 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:42:27.592 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:42:27.592 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:42:27.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:42:27.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:42:27.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:42:27.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:42:27.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:27.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:42:27.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:42:27.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:42:27.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:42:27.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:42:27.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:42:27.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:42:27.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:42:27.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:42:27.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:42:27.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:42:27.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:42:27.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:42:27.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:42:27.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:42:27.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:42:27.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:42:27.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:42:27.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:27.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:42:27.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:42:27.600 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:42:27.600 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:42:27.600 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:42:27.601 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_37 2022-04-10T01:42:27.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2022-04-10T01:42:27.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:42:27.712 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:42:27.716 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:42:27.716 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_37 2022-04-10T01:42:27.716 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:1 2022-04-10T01:42:27.716 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:42:27.717 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:42:27.717 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:1 cluster instance 2022-04-10T01:42:27.717 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:1 2022-04-10T01:42:27.717 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:42:27.717 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:42:27.718 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:42:27.718 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=1 2022-04-10T01:42:27.718 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 1 = cluster1:1 ']' 2022-04-10T01:42:27.718 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:42:27.719 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:42:27.719 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=1 2022-04-10T01:42:27.719 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:42:27.719 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:42:27.720 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:42:27.720 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:27.720 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:42:27.720 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:1 cluster1 2022-04-10T01:42:27.720 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:1 2022-04-10T01:42:27.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:42:27.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:42:27.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2022-04-10T01:42:27.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:1 2022-04-10T01:42:27.722 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:42:27.722 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:42:27.722 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:42:27.722 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:42:27.722 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 1 = cluster1:1 ']' 2022-04-10T01:42:27.723 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:42:27.723 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:42:27.723 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=1 2022-04-10T01:42:27.723 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=1 2022-04-10T01:42:27.724 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:42:27.724 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:42:27.724 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:27.724 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:42:27.724 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2022-04-10T01:42:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:42:27.735 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:42:27.735 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:42:27.735 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_37 2022-04-10T01:42:27.843 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:42:27.844 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:42:27.845 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:42:27.845 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:42:27.845 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:42:27.850 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:42:27.850 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:42:27.850 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:27.851 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_37 2022-04-10T01:42:27.851 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:42:27.851 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:27.851 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:27.851 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_37 2022-04-10T01:42:27.852 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:42:27.852 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:27.852 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:27.852 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:27.853 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:42:27.853 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:42:27.853 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_37 2022-04-10T01:42:27.853 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:42:27.853 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:42:27.854 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:27.854 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_37 2022-04-10T01:42:27.854 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:27.854 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:27.854 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:27.855 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:27.855 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:42:27.855 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:27.855 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:27.855 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:27.856 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-image_37.status 2022-04-10T01:42:27.856 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-image_37.status 2022-04-10T01:42:27.856 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:42:27.856 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_37.status 2022-04-10T01:42:27.856 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-image_37.status 2022-04-10T01:42:27.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-image_37.status 2022-04-10T01:42:27.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:42:27.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_37.status 2022-04-10T01:42:27.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror image_37 2022-04-10T01:42:27.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:27.858 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:27.858 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_37 2022-04-10T01:42:27.858 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:42:27.858 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:27.858 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:27.859 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:27.859 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_37 2022-04-10T01:42:28.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 227 2022-04-10T01:42:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_37 /tmp/tmp.rgwz74WspX/cluster2-mirror-image_37.status 2022-04-10T01:42:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_37 2022-04-10T01:42:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_37.status 2022-04-10T01:42:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:42:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_37 --format xml 2022-04-10T01:42:28.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:42:28.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:42:28.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=227 2022-04-10T01:42:28.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:42:28.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:42:28.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:42:28.837 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_37.status 2022-04-10T01:42:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_37 2022-04-10T01:42:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_37.status 2022-04-10T01:42:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2022-04-10T01:42:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:42:28.885 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:42:28.885 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:42:28.886 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:42:28.887 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:42:28.887 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:42:28.888 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:42:28.888 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 227 = '' 2022-04-10T01:42:28.888 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:42:28.889 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:42:29.197 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9373284542201065) > weight (0.25) 2022-04-10T01:42:29.197 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.937182682815994) > weight (0.25) 2022-04-10T01:42:29.197 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.5650037459626395) > weight (0.25) 2022-04-10T01:42:29.198 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.90777982124482) > weight (0.25) 2022-04-10T01:42:29.198 INFO:tasks.rbd_mirror_thrash:waiting for 110 secs before thrashing 2022-04-10T01:42:29.289 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_37.status 2022-04-10T01:42:29.289 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_37 2022-04-10T01:42:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_37.status 2022-04-10T01:42:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:29.291 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:29.291 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:29.291 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:29.291 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2022-04-10T01:42:29.291 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:42:29.337 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:42:29.338 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:42:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:42:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:42:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:42:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:42:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 227 = '' 2022-04-10T01:42:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:42:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:42:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_37.status 2022-04-10T01:42:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_37 2022-04-10T01:42:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_37.status 2022-04-10T01:42:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2022-04-10T01:42:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:42:30.184 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:42:30.186 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=227 2022-04-10T01:42:30.187 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 227 = 227 2022-04-10T01:42:30.187 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:30.187 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror image_37 snap 2022-04-10T01:42:30.187 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:30.187 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:30.188 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_37 2022-04-10T01:42:30.188 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:42:30.188 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:42:30.188 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:30.188 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:30.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:30.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:30.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:42:30.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:42:31.187 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/image_37@snap 2022-04-10T01:42:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_37': 2022-04-10T01:42:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:42:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:42:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 2 2022-04-10T01:42:31.230 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 10e4e5d8f1a6 2022-04-10T01:42:31.230 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.10e4e5d8f1a6 2022-04-10T01:42:31.230 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:42:31.230 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:42:31.231 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:42:31.231 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:42:31.231 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:34:48 2022 2022-04-10T01:42:31.232 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:34:48 2022 2022-04-10T01:42:31.232 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:34:48 2022 2022-04-10T01:42:31.232 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:42:31.232 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:42:31.233 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:42:31.233 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 6e506c1e-5b46-476c-afb6-a70d5f9862e5 2022-04-10T01:42:31.233 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:42:31.235 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:31.235 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror image_37 snap 2022-04-10T01:42:31.235 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:31.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_37 2022-04-10T01:42:31.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:42:31.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:42:31.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_37.export 2022-04-10T01:42:31.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_37.export 2022-04-10T01:42:31.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_37.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_37.export 2022-04-10T01:42:31.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export image_37@snap /tmp/tmp.rgwz74WspX/cluster2-mirror-image_37.export 2022-04-10T01:42:31.410 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:42:31.414 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export image_37@snap /tmp/tmp.rgwz74WspX/cluster1-mirror-image_37.export 2022-04-10T01:42:31.631 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:42:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-image_37.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_37.export 2022-04-10T01:42:31.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_37.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_37.export 2022-04-10T01:42:31.734 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:31.735 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:42:31.735 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_38 2022-04-10T01:42:31.735 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror image_38 snap 2022-04-10T01:42:31.735 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:31.736 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:31.736 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_38 2022-04-10T01:42:31.736 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:42:31.736 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create image_38@snap --debug-rbd=20 --debug-journaler=20 2022-04-10T01:42:33.592 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror image_38 2022-04-10T01:42:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_38 2022-04-10T01:42:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror image_38 started 2022-04-10T01:42:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_38 2022-04-10T01:42:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:42:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:42:33.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:33.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:33.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:33.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:33.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:42:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:42:34.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror image_38 started 2022-04-10T01:42:34.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:34.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:34.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_38 2022-04-10T01:42:34.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:42:34.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:42:34.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:42:34.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:34.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:34.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:34.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:34.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_38 2022-04-10T01:42:34.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:42:34.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:42:34.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:42:34.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:42:34.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:42:34.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:42:34.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:42:34.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:42:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:42:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:42:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:42:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:42:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:42:34.604 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:42:34.604 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_38 2022-04-10T01:42:34.604 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:42:34.604 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:42:34.604 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:42:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:42:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:42:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:42:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:42:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:42:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:42:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:42:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:42:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:42:34.607 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:42:34.607 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:42:34.607 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:42:34.607 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:42:34.608 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:34.608 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:42:34.608 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:42:34.608 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:42:34.608 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:42:34.609 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:42:34.609 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:42:34.609 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:42:34.609 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:42:34.609 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:42:34.610 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:42:34.610 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:42:34.610 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:42:34.610 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:42:34.610 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:42:34.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:42:34.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:42:34.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:42:34.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:42:34.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:34.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:42:34.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:42:34.616 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:42:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:42:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:42:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_38 2022-04-10T01:42:34.730 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2022-04-10T01:42:34.730 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:42:34.730 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:42:34.734 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:42:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_38 2022-04-10T01:42:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:1 2022-04-10T01:42:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:42:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:42:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:1 cluster instance 2022-04-10T01:42:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:1 2022-04-10T01:42:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:42:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:42:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:42:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=1 2022-04-10T01:42:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 1 = cluster1:1 ']' 2022-04-10T01:42:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:42:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:42:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=1 2022-04-10T01:42:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:42:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:42:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:42:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:42:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:1 cluster1 2022-04-10T01:42:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:1 2022-04-10T01:42:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:42:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:42:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2022-04-10T01:42:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:1 2022-04-10T01:42:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:42:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:42:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:42:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:42:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 1 = cluster1:1 ']' 2022-04-10T01:42:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:42:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:42:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=1 2022-04-10T01:42:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=1 2022-04-10T01:42:34.742 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:42:34.742 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:42:34.742 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:34.742 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:42:34.743 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2022-04-10T01:42:34.748 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:42:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:42:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:42:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_38 2022-04-10T01:42:34.863 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:42:34.863 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:42:34.864 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:42:34.864 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:42:34.864 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:42:34.868 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:42:34.869 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:42:34.869 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:34.869 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_38 2022-04-10T01:42:34.869 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:42:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_38 2022-04-10T01:42:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:42:34.871 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:34.871 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:34.871 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:34.871 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:42:34.871 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:42:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_38 2022-04-10T01:42:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:42:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:42:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_38 2022-04-10T01:42:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:34.874 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:42:34.874 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:34.874 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:34.874 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:34.874 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-image_38.status 2022-04-10T01:42:34.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-image_38.status 2022-04-10T01:42:34.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:42:34.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_38.status 2022-04-10T01:42:34.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-image_38.status 2022-04-10T01:42:34.876 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-image_38.status 2022-04-10T01:42:34.876 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:42:34.876 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_38.status 2022-04-10T01:42:34.876 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror image_38 2022-04-10T01:42:34.877 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:34.877 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:34.877 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_38 2022-04-10T01:42:34.877 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:42:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_38 2022-04-10T01:42:35.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 230 2022-04-10T01:42:35.607 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_38 /tmp/tmp.rgwz74WspX/cluster2-mirror-image_38.status 2022-04-10T01:42:35.607 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:35.607 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:35.607 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_38 2022-04-10T01:42:35.608 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_38.status 2022-04-10T01:42:35.608 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:42:35.608 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:35.608 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:35.609 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:35.609 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_38 --format xml 2022-04-10T01:42:35.609 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:42:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:42:35.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=230 2022-04-10T01:42:35.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:42:35.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:42:35.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:42:35.855 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_38.status 2022-04-10T01:42:35.855 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:35.855 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:35.856 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_38 2022-04-10T01:42:35.856 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_38.status 2022-04-10T01:42:35.856 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:35.856 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2022-04-10T01:42:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:42:35.904 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:42:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:42:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:42:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:42:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:42:35.907 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:42:35.907 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 230 = '' 2022-04-10T01:42:35.908 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:42:35.908 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:42:36.308 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_38.status 2022-04-10T01:42:36.309 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:36.309 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:36.309 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_38 2022-04-10T01:42:36.309 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_38.status 2022-04-10T01:42:36.310 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:36.310 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:36.310 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:36.310 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:36.311 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2022-04-10T01:42:36.311 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:42:36.357 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:42:36.357 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:42:36.359 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:42:36.359 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:42:36.359 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:42:36.360 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:42:36.360 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 230 = '' 2022-04-10T01:42:36.360 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:42:36.361 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:42:37.161 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_38.status 2022-04-10T01:42:37.162 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:37.162 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:37.162 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_38 2022-04-10T01:42:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_38.status 2022-04-10T01:42:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2022-04-10T01:42:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:42:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:42:37.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=230 2022-04-10T01:42:37.207 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 230 = 230 2022-04-10T01:42:37.207 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:37.207 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror image_38 snap 2022-04-10T01:42:37.207 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:37.208 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:37.208 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_38 2022-04-10T01:42:37.208 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:42:37.208 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:42:37.209 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:37.209 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:37.209 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:37.209 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:37.210 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:42:37.210 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:42:38.207 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/image_38@snap 2022-04-10T01:42:38.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_38': 2022-04-10T01:42:38.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:42:38.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:42:38.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 2 2022-04-10T01:42:38.254 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 11331a0c1e44 2022-04-10T01:42:38.254 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.11331a0c1e44 2022-04-10T01:42:38.254 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:42:38.254 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:42:38.255 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:42:38.255 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:42:38.255 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:34:58 2022 2022-04-10T01:42:38.255 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:34:58 2022 2022-04-10T01:42:38.256 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:34:58 2022 2022-04-10T01:42:38.256 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:42:38.256 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:42:38.256 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:42:38.257 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 643d8009-5a9c-482d-a7cf-86349b296923 2022-04-10T01:42:38.257 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:42:38.258 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:38.259 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror image_38 snap 2022-04-10T01:42:38.259 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:38.259 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_38 2022-04-10T01:42:38.259 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:42:38.260 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:42:38.260 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_38.export 2022-04-10T01:42:38.260 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_38.export 2022-04-10T01:42:38.260 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_38.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_38.export 2022-04-10T01:42:38.261 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export image_38@snap /tmp/tmp.rgwz74WspX/cluster2-mirror-image_38.export 2022-04-10T01:42:38.444 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:42:38.447 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export image_38@snap /tmp/tmp.rgwz74WspX/cluster1-mirror-image_38.export 2022-04-10T01:42:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:42:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-image_38.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_38.export 2022-04-10T01:42:38.734 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_38.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_38.export 2022-04-10T01:42:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:42:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_39 2022-04-10T01:42:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror image_39 snap 2022-04-10T01:42:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:38.773 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_39 2022-04-10T01:42:38.773 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:42:38.773 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create image_39@snap --debug-rbd=20 --debug-journaler=20 2022-04-10T01:42:40.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror image_39 2022-04-10T01:42:40.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:40.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:40.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_39 2022-04-10T01:42:40.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:40.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:40.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:40.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:40.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror image_39 started 2022-04-10T01:42:40.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:40.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:40.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_39 2022-04-10T01:42:40.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:42:40.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:42:40.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:40.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:40.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:40.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:40.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:42:40.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:42:41.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror image_39 started 2022-04-10T01:42:41.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:41.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:41.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_39 2022-04-10T01:42:41.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:42:41.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:42:41.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:42:41.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:41.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:41.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:41.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_39 2022-04-10T01:42:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:42:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:42:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:42:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:42:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:42:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:42:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:42:41.543 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:42:41.543 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:41.543 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:42:41.543 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:42:41.544 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:42:41.544 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:42:41.544 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:42:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:42:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_39 2022-04-10T01:42:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:42:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:42:41.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:42:41.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:42:41.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:42:41.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:42:41.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:42:41.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:42:41.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:42:41.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:42:41.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:42:41.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:42:41.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:42:41.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:42:41.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:42:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:42:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:42:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:42:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:42:41.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:42:41.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:42:41.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:42:41.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:42:41.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:42:41.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:42:41.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:42:41.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:42:41.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:42:41.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:42:41.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:42:41.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:42:41.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:42:41.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:42:41.556 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:42:41.556 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:41.556 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:42:41.556 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:42:41.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:42:41.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:42:41.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:42:41.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_39 2022-04-10T01:42:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2022-04-10T01:42:41.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:42:41.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:42:41.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:42:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_39 2022-04-10T01:42:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:1 2022-04-10T01:42:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:42:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:42:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:1 cluster instance 2022-04-10T01:42:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:1 2022-04-10T01:42:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:42:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:42:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:42:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=1 2022-04-10T01:42:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 1 = cluster1:1 ']' 2022-04-10T01:42:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:42:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:42:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=1 2022-04-10T01:42:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:42:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:42:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:42:41.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:41.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:42:41.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:1 cluster1 2022-04-10T01:42:41.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:1 2022-04-10T01:42:41.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:42:41.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:42:41.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2022-04-10T01:42:41.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:1 2022-04-10T01:42:41.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:42:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:42:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:42:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:42:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 1 = cluster1:1 ']' 2022-04-10T01:42:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:42:41.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:42:41.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=1 2022-04-10T01:42:41.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=1 2022-04-10T01:42:41.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:42:41.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:42:41.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:41.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:42:41.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2022-04-10T01:42:41.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:42:41.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:42:41.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:42:41.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_39 2022-04-10T01:42:41.799 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2022-04-10T01:42:41.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:42:41.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:42:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:42:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_39 2022-04-10T01:42:41.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:2 2022-04-10T01:42:41.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:42:41.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:42:41.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:2 cluster instance 2022-04-10T01:42:41.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:2 2022-04-10T01:42:41.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:42:41.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:42:41.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:42:41.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=2 2022-04-10T01:42:41.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 2 = cluster1:2 ']' 2022-04-10T01:42:41.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:42:41.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:42:41.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=2 2022-04-10T01:42:41.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:42:41.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:42:41.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:42:41.809 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:41.809 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:42:41.810 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:2 cluster1 2022-04-10T01:42:41.810 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:2 2022-04-10T01:42:41.810 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:42:41.810 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:42:41.811 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2022-04-10T01:42:41.811 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:2 2022-04-10T01:42:41.811 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:42:41.811 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:42:41.811 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:42:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:42:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 2 = cluster1:2 ']' 2022-04-10T01:42:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:42:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:42:41.813 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=2 2022-04-10T01:42:41.813 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=2 2022-04-10T01:42:41.813 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:42:41.813 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:42:41.813 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:41.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:42:41.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2022-04-10T01:42:41.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:42:41.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:42:41.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:42:41.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_39 2022-04-10T01:42:41.933 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:42:41.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:42:41.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:42:41.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:42:41.935 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:42:41.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:42:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:42:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_39 2022-04-10T01:42:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:42:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:41.941 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:41.941 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_39 2022-04-10T01:42:41.941 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:42:41.941 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:41.942 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:41.942 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:41.942 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:42:41.942 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:42:41.943 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_39 2022-04-10T01:42:41.943 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:42:41.943 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:42:41.943 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:41.943 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_39 2022-04-10T01:42:41.944 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:41.944 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:41.944 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:41.944 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:41.945 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:42:41.945 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:41.945 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:41.945 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:41.945 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-image_39.status 2022-04-10T01:42:41.946 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-image_39.status 2022-04-10T01:42:41.946 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:42:41.946 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_39.status 2022-04-10T01:42:41.946 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-image_39.status 2022-04-10T01:42:41.947 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-image_39.status 2022-04-10T01:42:41.947 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:42:41.947 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_39.status 2022-04-10T01:42:41.947 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror image_39 2022-04-10T01:42:41.947 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_39 2022-04-10T01:42:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:42:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:41.949 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:41.949 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:41.949 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_39 2022-04-10T01:42:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 233 2022-04-10T01:42:42.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_39 /tmp/tmp.rgwz74WspX/cluster2-mirror-image_39.status 2022-04-10T01:42:42.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:42.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:42.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_39 2022-04-10T01:42:42.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_39.status 2022-04-10T01:42:42.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:42:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_39 --format xml 2022-04-10T01:42:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:42:42.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:42:42.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=233 2022-04-10T01:42:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:42:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:42:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:42:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_39.status 2022-04-10T01:42:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_39 2022-04-10T01:42:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_39.status 2022-04-10T01:42:42.872 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:42.872 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:42.872 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:42.872 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2022-04-10T01:42:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:42:42.918 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:42:42.919 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:42:42.920 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:42:42.920 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:42:42.920 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:42:42.921 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:42:42.921 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 233 = '' 2022-04-10T01:42:42.922 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:42:42.922 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:42:43.322 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_39.status 2022-04-10T01:42:43.323 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:43.323 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:43.323 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_39 2022-04-10T01:42:43.324 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_39.status 2022-04-10T01:42:43.324 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:43.324 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:43.324 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:43.324 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:43.325 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2022-04-10T01:42:43.325 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:42:43.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:42:43.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:42:43.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:42:43.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:42:43.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:42:43.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:42:43.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 233 = '' 2022-04-10T01:42:43.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:42:43.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:42:44.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_39.status 2022-04-10T01:42:44.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:44.380 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:44.380 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_39 2022-04-10T01:42:44.380 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_39.status 2022-04-10T01:42:44.380 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:44.380 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2022-04-10T01:42:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:42:44.422 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:42:44.425 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=233 2022-04-10T01:42:44.425 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 233 = 233 2022-04-10T01:42:44.426 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:44.426 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror image_39 snap 2022-04-10T01:42:44.426 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:44.426 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_39 2022-04-10T01:42:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:42:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:42:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:44.428 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:44.428 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:44.428 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:44.428 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:42:44.428 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:42:45.425 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/image_39@snap 2022-04-10T01:42:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_39': 2022-04-10T01:42:45.470 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:42:45.470 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:42:45.470 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 2 2022-04-10T01:42:45.470 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 118d5e8fb3eb 2022-04-10T01:42:45.471 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.118d5e8fb3eb 2022-04-10T01:42:45.471 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:42:45.471 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:42:45.471 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:42:45.471 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:42:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:35:05 2022 2022-04-10T01:42:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:35:05 2022 2022-04-10T01:42:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:35:05 2022 2022-04-10T01:42:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:42:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:42:45.473 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:42:45.473 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: e2bbf2e7-46b6-4dd8-b6e1-9dea8cf3b0b4 2022-04-10T01:42:45.473 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:42:45.474 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror image_39 snap 2022-04-10T01:42:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_39 2022-04-10T01:42:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:42:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:42:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_39.export 2022-04-10T01:42:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_39.export 2022-04-10T01:42:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_39.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_39.export 2022-04-10T01:42:45.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export image_39@snap /tmp/tmp.rgwz74WspX/cluster2-mirror-image_39.export 2022-04-10T01:42:45.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:42:45.659 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export image_39@snap /tmp/tmp.rgwz74WspX/cluster1-mirror-image_39.export 2022-04-10T01:42:45.886 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:42:45.890 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-image_39.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_39.export 2022-04-10T01:42:45.953 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_39.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_39.export 2022-04-10T01:42:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:42:45.992 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_40 2022-04-10T01:42:45.992 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror image_40 snap 2022-04-10T01:42:45.992 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:45.992 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:45.993 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_40 2022-04-10T01:42:45.993 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:42:45.993 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create image_40@snap --debug-rbd=20 --debug-journaler=20 2022-04-10T01:42:47.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror image_40 2022-04-10T01:42:47.621 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:47.621 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:47.621 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_40 2022-04-10T01:42:47.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:47.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:47.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:47.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:47.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror image_40 started 2022-04-10T01:42:47.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:47.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:47.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_40 2022-04-10T01:42:47.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:42:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:42:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:47.625 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:47.625 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:42:47.625 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:42:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror image_40 started 2022-04-10T01:42:48.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:48.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:48.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_40 2022-04-10T01:42:48.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:42:48.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:42:48.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:42:48.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:48.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:48.625 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:48.625 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:48.625 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_40 2022-04-10T01:42:48.625 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:42:48.625 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:42:48.626 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:42:48.626 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:42:48.626 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:42:48.626 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:42:48.626 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:42:48.627 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:42:48.627 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:48.627 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:42:48.627 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:42:48.627 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:42:48.631 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:42:48.631 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:42:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:42:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_40 2022-04-10T01:42:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:42:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:42:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:42:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:42:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:42:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:42:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:42:48.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:42:48.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:42:48.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:42:48.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:42:48.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:42:48.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:42:48.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:42:48.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:42:48.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:42:48.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:48.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:42:48.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:42:48.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:42:48.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:42:48.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:42:48.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:42:48.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:42:48.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:42:48.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:42:48.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:42:48.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:42:48.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:42:48.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:42:48.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:42:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:42:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:42:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:42:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:42:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:48.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:42:48.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:42:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:42:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:42:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:42:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_40 2022-04-10T01:42:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2022-04-10T01:42:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:42:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:42:48.762 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:42:48.763 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_40 2022-04-10T01:42:48.763 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:1 2022-04-10T01:42:48.763 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:42:48.763 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:42:48.764 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:1 cluster instance 2022-04-10T01:42:48.764 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:1 2022-04-10T01:42:48.764 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:42:48.764 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:42:48.764 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:42:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=1 2022-04-10T01:42:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 1 = cluster1:1 ']' 2022-04-10T01:42:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:42:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:42:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=1 2022-04-10T01:42:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:42:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:42:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:42:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:42:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:1 cluster1 2022-04-10T01:42:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:1 2022-04-10T01:42:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:42:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:42:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2022-04-10T01:42:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:1 2022-04-10T01:42:48.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:42:48.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:42:48.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:42:48.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:42:48.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 1 = cluster1:1 ']' 2022-04-10T01:42:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:42:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:42:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=1 2022-04-10T01:42:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=1 2022-04-10T01:42:48.770 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:42:48.770 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:42:48.770 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:48.770 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:42:48.770 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2022-04-10T01:42:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:42:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:42:48.778 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:42:48.778 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_40 2022-04-10T01:42:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:42:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:42:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:42:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:42:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:42:48.901 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:42:48.901 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:42:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_40 2022-04-10T01:42:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:42:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_40 2022-04-10T01:42:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:42:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:48.904 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:48.904 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:42:48.904 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:42:48.904 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_40 2022-04-10T01:42:48.904 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:42:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:42:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_40 2022-04-10T01:42:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:42:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:48.907 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:48.907 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-image_40.status 2022-04-10T01:42:48.907 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-image_40.status 2022-04-10T01:42:48.907 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:42:48.907 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_40.status 2022-04-10T01:42:48.908 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-image_40.status 2022-04-10T01:42:48.908 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-image_40.status 2022-04-10T01:42:48.908 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:42:48.908 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_40.status 2022-04-10T01:42:48.908 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror image_40 2022-04-10T01:42:48.909 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:48.909 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:48.909 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_40 2022-04-10T01:42:48.909 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:42:48.909 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:48.910 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:48.910 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:48.910 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_40 2022-04-10T01:42:49.621 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 236 2022-04-10T01:42:49.631 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_40 /tmp/tmp.rgwz74WspX/cluster2-mirror-image_40.status 2022-04-10T01:42:49.631 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:49.631 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:49.632 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_40 2022-04-10T01:42:49.632 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_40.status 2022-04-10T01:42:49.632 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:42:49.632 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:49.632 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:49.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:49.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_40 --format xml 2022-04-10T01:42:49.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:42:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:42:49.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=236 2022-04-10T01:42:49.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:42:49.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:42:49.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:42:49.877 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_40.status 2022-04-10T01:42:49.877 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:49.877 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_40 2022-04-10T01:42:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_40.status 2022-04-10T01:42:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2022-04-10T01:42:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:42:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:42:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:42:49.922 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:42:49.922 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:42:49.923 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:42:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:42:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 236 = '' 2022-04-10T01:42:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:42:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:42:50.325 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_40.status 2022-04-10T01:42:50.325 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:50.325 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:50.326 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_40 2022-04-10T01:42:50.326 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_40.status 2022-04-10T01:42:50.326 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:50.326 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:50.327 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:50.327 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:50.327 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2022-04-10T01:42:50.327 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:42:50.373 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:42:50.373 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:42:50.375 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:42:50.375 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:42:50.375 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:42:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:42:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 236 = '' 2022-04-10T01:42:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:42:50.377 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:42:51.177 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_40.status 2022-04-10T01:42:51.178 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:51.178 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:51.178 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_40 2022-04-10T01:42:51.178 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_40.status 2022-04-10T01:42:51.178 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:51.179 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:51.179 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:51.179 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:51.179 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2022-04-10T01:42:51.180 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:42:51.220 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:42:51.223 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=236 2022-04-10T01:42:51.223 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 236 = 236 2022-04-10T01:42:51.223 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:51.223 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror image_40 snap 2022-04-10T01:42:51.224 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:51.224 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:51.224 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_40 2022-04-10T01:42:51.224 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:42:51.224 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:42:51.225 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:51.225 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:51.225 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:51.225 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:51.225 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:42:51.226 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:42:52.223 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/image_40@snap 2022-04-10T01:42:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_40': 2022-04-10T01:42:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:42:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:42:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 2 2022-04-10T01:42:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 12a75031885c 2022-04-10T01:42:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.12a75031885c 2022-04-10T01:42:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:42:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:42:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:42:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:42:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:35:16 2022 2022-04-10T01:42:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:35:16 2022 2022-04-10T01:42:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:35:16 2022 2022-04-10T01:42:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:42:52.270 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:42:52.270 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:42:52.270 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 778e4963-897c-4526-9fc0-0a90b3039381 2022-04-10T01:42:52.270 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:42:52.272 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:52.272 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror image_40 snap 2022-04-10T01:42:52.272 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:52.272 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_40 2022-04-10T01:42:52.273 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:42:52.273 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:42:52.273 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_40.export 2022-04-10T01:42:52.273 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_40.export 2022-04-10T01:42:52.274 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_40.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_40.export 2022-04-10T01:42:52.274 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export image_40@snap /tmp/tmp.rgwz74WspX/cluster2-mirror-image_40.export 2022-04-10T01:42:52.455 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:42:52.459 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export image_40@snap /tmp/tmp.rgwz74WspX/cluster1-mirror-image_40.export 2022-04-10T01:42:52.715 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:42:52.718 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-image_40.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_40.export 2022-04-10T01:42:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_40.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_40.export 2022-04-10T01:42:52.819 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:52.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:42:52.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_41 2022-04-10T01:42:52.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror image_41 snap 2022-04-10T01:42:52.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:52.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:52.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_41 2022-04-10T01:42:52.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:42:52.823 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create image_41@snap --debug-rbd=20 --debug-journaler=20 2022-04-10T01:42:54.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror image_41 2022-04-10T01:42:54.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:54.540 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:54.541 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_41 2022-04-10T01:42:54.541 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:54.541 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:54.541 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:54.541 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:54.542 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror image_41 started 2022-04-10T01:42:54.542 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:54.542 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:54.542 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_41 2022-04-10T01:42:54.543 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:42:54.543 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:42:54.543 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:54.543 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:54.543 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:54.544 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:54.544 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:42:54.544 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:42:55.542 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror image_41 started 2022-04-10T01:42:55.542 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:55.543 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:55.543 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_41 2022-04-10T01:42:55.543 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:42:55.543 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:42:55.543 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:42:55.544 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:55.544 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:55.544 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:55.544 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:55.544 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_41 2022-04-10T01:42:55.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:42:55.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:42:55.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:42:55.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:42:55.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:42:55.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:42:55.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:42:55.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:42:55.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:42:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:42:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:42:55.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:42:55.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:42:55.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:42:55.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_41 2022-04-10T01:42:55.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:42:55.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:42:55.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:42:55.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:42:55.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:42:55.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:42:55.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:42:55.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:42:55.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:42:55.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:42:55.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:42:55.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:42:55.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:42:55.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:42:55.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:42:55.556 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:42:55.556 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:55.556 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:42:55.556 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:42:55.557 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:42:55.557 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:42:55.557 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:42:55.557 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:42:55.557 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:42:55.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:42:55.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:42:55.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:42:55.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:42:55.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:42:55.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:42:55.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:42:55.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:42:55.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:42:55.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:42:55.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:42:55.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:55.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:42:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:42:55.566 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:42:55.567 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:42:55.567 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:42:55.567 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_41 2022-04-10T01:42:55.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2022-04-10T01:42:55.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:42:55.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:42:55.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:42:55.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_41 2022-04-10T01:42:55.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:1 2022-04-10T01:42:55.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:42:55.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:42:55.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:1 cluster instance 2022-04-10T01:42:55.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:1 2022-04-10T01:42:55.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:42:55.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:42:55.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:42:55.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=1 2022-04-10T01:42:55.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 1 = cluster1:1 ']' 2022-04-10T01:42:55.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:42:55.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:42:55.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=1 2022-04-10T01:42:55.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:42:55.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:42:55.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:42:55.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:55.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:42:55.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:1 cluster1 2022-04-10T01:42:55.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:1 2022-04-10T01:42:55.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:42:55.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:42:55.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2022-04-10T01:42:55.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:1 2022-04-10T01:42:55.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:42:55.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:42:55.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:42:55.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:42:55.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 1 = cluster1:1 ']' 2022-04-10T01:42:55.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:42:55.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:42:55.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=1 2022-04-10T01:42:55.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=1 2022-04-10T01:42:55.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:42:55.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:42:55.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:55.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:42:55.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2022-04-10T01:42:55.698 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:42:55.698 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:42:55.698 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:42:55.698 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_41 2022-04-10T01:42:55.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2022-04-10T01:42:55.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:42:55.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:42:55.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:42:55.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_41 2022-04-10T01:42:55.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:2 2022-04-10T01:42:55.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:42:55.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:42:55.809 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:2 cluster instance 2022-04-10T01:42:55.809 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:2 2022-04-10T01:42:55.809 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:42:55.809 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:42:55.810 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:42:55.810 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=2 2022-04-10T01:42:55.810 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 2 = cluster1:2 ']' 2022-04-10T01:42:55.810 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:42:55.810 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:42:55.811 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=2 2022-04-10T01:42:55.811 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:42:55.811 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:42:55.811 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:42:55.811 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:55.812 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:42:55.812 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:2 cluster1 2022-04-10T01:42:55.812 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:2 2022-04-10T01:42:55.812 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:42:55.812 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:42:55.813 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2022-04-10T01:42:55.813 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:2 2022-04-10T01:42:55.813 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:42:55.813 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:42:55.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:42:55.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:42:55.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 2 = cluster1:2 ']' 2022-04-10T01:42:55.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:42:55.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:42:55.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=2 2022-04-10T01:42:55.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=2 2022-04-10T01:42:55.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:42:55.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:42:55.816 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:55.816 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:42:55.816 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2022-04-10T01:42:55.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:42:55.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:42:55.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:42:55.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_41 2022-04-10T01:42:55.928 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2022-04-10T01:42:55.928 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:42:55.928 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:42:55.932 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:42:55.932 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_41 2022-04-10T01:42:55.933 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:3 2022-04-10T01:42:55.933 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:42:55.933 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:42:55.933 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:3 cluster instance 2022-04-10T01:42:55.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:3 2022-04-10T01:42:55.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:42:55.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:42:55.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:42:55.935 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=3 2022-04-10T01:42:55.935 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 3 = cluster1:3 ']' 2022-04-10T01:42:55.935 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:42:55.935 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:42:55.935 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=3 2022-04-10T01:42:55.936 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=3 2022-04-10T01:42:55.936 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:42:55.936 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:42:55.936 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:55.936 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:42:55.937 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:3 cluster1 2022-04-10T01:42:55.937 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:3 2022-04-10T01:42:55.937 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:42:55.937 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:42:55.937 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2022-04-10T01:42:55.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:3 2022-04-10T01:42:55.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:42:55.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:42:55.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:42:55.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=3 2022-04-10T01:42:55.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 3 = cluster1:3 ']' 2022-04-10T01:42:55.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:42:55.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:42:55.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=3 2022-04-10T01:42:55.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=3 2022-04-10T01:42:55.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:42:55.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:42:55.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:55.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:42:55.941 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2022-04-10T01:42:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:42:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:42:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:42:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_41 2022-04-10T01:42:56.061 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:42:56.061 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:42:56.062 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:42:56.062 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:42:56.063 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:42:56.066 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:42:56.066 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:42:56.066 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:56.067 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_41 2022-04-10T01:42:56.067 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:42:56.067 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:56.067 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:56.067 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_41 2022-04-10T01:42:56.068 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:42:56.068 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:56.068 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:56.068 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:56.069 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:42:56.069 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:42:56.069 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_41 2022-04-10T01:42:56.069 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:42:56.069 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:42:56.070 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:56.070 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_41 2022-04-10T01:42:56.070 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:56.070 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:56.071 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:56.071 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:56.071 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:42:56.071 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:56.071 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:56.072 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:56.072 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-image_41.status 2022-04-10T01:42:56.072 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-image_41.status 2022-04-10T01:42:56.072 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:42:56.072 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_41.status 2022-04-10T01:42:56.073 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-image_41.status 2022-04-10T01:42:56.073 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-image_41.status 2022-04-10T01:42:56.073 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:42:56.074 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_41.status 2022-04-10T01:42:56.075 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror image_41 2022-04-10T01:42:56.075 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:56.075 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:56.075 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_41 2022-04-10T01:42:56.076 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:42:56.076 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:56.076 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:56.076 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:56.076 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_41 2022-04-10T01:42:56.631 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 239 2022-04-10T01:42:56.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_41 /tmp/tmp.rgwz74WspX/cluster2-mirror-image_41.status 2022-04-10T01:42:56.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:56.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:56.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_41 2022-04-10T01:42:56.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_41.status 2022-04-10T01:42:56.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:42:56.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:56.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:56.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:56.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_41 --format xml 2022-04-10T01:42:56.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:42:56.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:42:56.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=239 2022-04-10T01:42:56.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:42:56.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:42:56.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:42:56.886 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_41.status 2022-04-10T01:42:56.886 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:56.886 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_41 2022-04-10T01:42:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_41.status 2022-04-10T01:42:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:56.888 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:56.888 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:56.888 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2022-04-10T01:42:56.888 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:42:56.927 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:42:56.927 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:42:56.928 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:42:56.929 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:42:56.929 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:42:56.930 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:42:56.930 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 239 = '' 2022-04-10T01:42:56.930 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:42:56.931 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:42:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_41.status 2022-04-10T01:42:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_41 2022-04-10T01:42:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_41.status 2022-04-10T01:42:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:57.333 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:57.333 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:57.333 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:57.333 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2022-04-10T01:42:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:42:57.376 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:42:57.376 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:42:57.377 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:42:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:42:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:42:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:42:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 239 = '' 2022-04-10T01:42:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:42:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:42:58.180 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_41.status 2022-04-10T01:42:58.180 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:58.181 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:58.181 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_41 2022-04-10T01:42:58.181 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_41.status 2022-04-10T01:42:58.181 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:58.182 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:58.182 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:58.182 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:58.182 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2022-04-10T01:42:58.182 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:42:58.222 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:42:58.225 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=239 2022-04-10T01:42:58.225 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 239 = 239 2022-04-10T01:42:58.225 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:58.226 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror image_41 snap 2022-04-10T01:42:58.226 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:58.226 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:58.226 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_41 2022-04-10T01:42:58.227 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:42:58.227 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:42:58.227 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:42:58.227 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:42:58.228 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:42:58.228 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:58.228 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:42:58.228 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:42:59.226 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/image_41@snap 2022-04-10T01:42:59.268 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_41': 2022-04-10T01:42:59.268 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:42:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:42:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 2 2022-04-10T01:42:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 10e437fadbb8 2022-04-10T01:42:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.10e437fadbb8 2022-04-10T01:42:59.270 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:42:59.270 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:42:59.270 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:42:59.270 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:42:59.270 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:35:23 2022 2022-04-10T01:42:59.271 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:35:23 2022 2022-04-10T01:42:59.271 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:35:23 2022 2022-04-10T01:42:59.271 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:42:59.271 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:42:59.272 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:42:59.272 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: af549731-3fee-45da-9830-6e51d5343406 2022-04-10T01:42:59.272 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:42:59.273 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:59.274 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror image_41 snap 2022-04-10T01:42:59.274 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:59.274 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_41 2022-04-10T01:42:59.274 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:42:59.274 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:42:59.275 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_41.export 2022-04-10T01:42:59.275 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_41.export 2022-04-10T01:42:59.275 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_41.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_41.export 2022-04-10T01:42:59.275 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export image_41@snap /tmp/tmp.rgwz74WspX/cluster2-mirror-image_41.export 2022-04-10T01:42:59.450 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:42:59.453 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export image_41@snap /tmp/tmp.rgwz74WspX/cluster1-mirror-image_41.export 2022-04-10T01:42:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:42:59.702 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-image_41.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_41.export 2022-04-10T01:42:59.766 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_41.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_41.export 2022-04-10T01:42:59.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:42:59.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:42:59.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_42 2022-04-10T01:42:59.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror image_42 snap 2022-04-10T01:42:59.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:42:59.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:42:59.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_42 2022-04-10T01:42:59.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:42:59.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create image_42@snap --debug-rbd=20 --debug-journaler=20 2022-04-10T01:43:01.649 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror image_42 2022-04-10T01:43:01.649 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:01.649 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:01.650 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_42 2022-04-10T01:43:01.650 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:01.650 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:01.650 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror image_42 started 2022-04-10T01:43:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:01.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_42 2022-04-10T01:43:01.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:43:01.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:43:01.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:01.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:01.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:01.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:01.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:43:01.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:43:02.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror image_42 started 2022-04-10T01:43:02.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:02.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:02.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_42 2022-04-10T01:43:02.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:43:02.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:43:02.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:43:02.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:02.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:02.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:02.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:02.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_42 2022-04-10T01:43:02.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:43:02.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:43:02.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:43:02.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:43:02.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:43:02.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:43:02.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:43:02.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:43:02.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:02.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:43:02.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:43:02.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:43:02.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:43:02.660 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:43:02.661 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:43:02.661 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_42 2022-04-10T01:43:02.661 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:43:02.661 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:43:02.662 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:43:02.662 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:43:02.662 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:43:02.662 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:43:02.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:43:02.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:43:02.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:43:02.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:43:02.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:43:02.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:02.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:43:02.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:43:02.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:43:02.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:43:02.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:02.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:43:02.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:43:02.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:43:02.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:02.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:43:02.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:43:02.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:43:02.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:43:02.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:43:02.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:02.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:43:02.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:43:02.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:43:02.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:43:02.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:43:02.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:43:02.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:43:02.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:02.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:02.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:43:02.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:43:02.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:43:02.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:43:02.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:43:02.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_42 2022-04-10T01:43:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2022-04-10T01:43:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:43:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:43:02.791 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:43:02.791 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_42 2022-04-10T01:43:02.792 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:1 2022-04-10T01:43:02.792 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:43:02.792 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:43:02.792 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:1 cluster instance 2022-04-10T01:43:02.793 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:1 2022-04-10T01:43:02.793 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:43:02.793 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:43:02.793 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:43:02.793 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=1 2022-04-10T01:43:02.794 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 1 = cluster1:1 ']' 2022-04-10T01:43:02.794 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:43:02.794 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:02.794 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=1 2022-04-10T01:43:02.795 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:43:02.795 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:43:02.795 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:43:02.795 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:02.795 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:43:02.796 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:1 cluster1 2022-04-10T01:43:02.796 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:1 2022-04-10T01:43:02.796 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:02.796 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:43:02.797 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2022-04-10T01:43:02.797 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:1 2022-04-10T01:43:02.797 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:43:02.797 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:43:02.797 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:02.798 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:43:02.798 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 1 = cluster1:1 ']' 2022-04-10T01:43:02.798 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:43:02.798 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:43:02.799 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=1 2022-04-10T01:43:02.799 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=1 2022-04-10T01:43:02.799 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:43:02.799 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:02.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:02.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:43:02.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2022-04-10T01:43:02.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:43:02.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:43:02.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:43:02.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_42 2022-04-10T01:43:02.911 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2022-04-10T01:43:02.942 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:43:02.943 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:43:02.947 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:43:02.947 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_42 2022-04-10T01:43:02.947 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:2 2022-04-10T01:43:02.947 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:43:02.948 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:43:02.948 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:2 cluster instance 2022-04-10T01:43:02.948 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:2 2022-04-10T01:43:02.948 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:43:02.948 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:43:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:43:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=2 2022-04-10T01:43:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 2 = cluster1:2 ']' 2022-04-10T01:43:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:43:02.950 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:02.950 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=2 2022-04-10T01:43:02.951 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:43:02.951 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:43:02.951 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:43:02.952 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:02.952 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:43:02.952 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:2 cluster1 2022-04-10T01:43:02.953 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:2 2022-04-10T01:43:02.953 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:02.953 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:43:02.954 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2022-04-10T01:43:02.954 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:2 2022-04-10T01:43:02.954 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:43:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:43:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:43:02.956 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 2 = cluster1:2 ']' 2022-04-10T01:43:02.956 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:43:02.956 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:43:02.957 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=2 2022-04-10T01:43:02.957 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=2 2022-04-10T01:43:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:43:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:43:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2022-04-10T01:43:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:43:02.960 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:43:02.960 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:43:02.960 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_42 2022-04-10T01:43:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:43:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:43:03.047 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:43:03.047 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:43:03.048 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:43:03.055 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:43:03.056 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:43:03.056 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:03.057 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_42 2022-04-10T01:43:03.057 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:43:03.058 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:03.058 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:03.058 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_42 2022-04-10T01:43:03.059 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:43:03.059 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:03.060 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:03.060 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:03.061 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:43:03.061 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:43:03.061 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_42 2022-04-10T01:43:03.062 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:43:03.062 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:43:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_42 2022-04-10T01:43:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:03.065 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:03.065 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:43:03.065 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:03.066 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:03.066 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:03.066 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-image_42.status 2022-04-10T01:43:03.067 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-image_42.status 2022-04-10T01:43:03.067 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:43:03.067 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_42.status 2022-04-10T01:43:03.068 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-image_42.status 2022-04-10T01:43:03.068 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-image_42.status 2022-04-10T01:43:03.069 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:43:03.069 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_42.status 2022-04-10T01:43:03.069 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror image_42 2022-04-10T01:43:03.069 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:03.070 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:03.070 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_42 2022-04-10T01:43:03.070 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:43:03.070 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:03.071 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:03.071 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:03.071 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_42 2022-04-10T01:43:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 242 2022-04-10T01:43:03.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_42 /tmp/tmp.rgwz74WspX/cluster2-mirror-image_42.status 2022-04-10T01:43:03.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:03.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:03.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_42 2022-04-10T01:43:03.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_42.status 2022-04-10T01:43:03.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:43:03.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:03.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:03.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_42 --format xml 2022-04-10T01:43:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:43:03.706 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:43:03.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=242 2022-04-10T01:43:03.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:43:03.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:43:03.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:43:03.910 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_42.status 2022-04-10T01:43:03.910 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:03.911 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:03.911 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_42 2022-04-10T01:43:03.911 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_42.status 2022-04-10T01:43:03.911 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:03.911 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:03.912 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:03.912 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:03.912 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2022-04-10T01:43:03.912 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:43:03.955 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:43:03.955 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:43:03.956 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:43:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:43:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:43:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:43:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 242 = '' 2022-04-10T01:43:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:43:03.959 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:43:04.359 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_42.status 2022-04-10T01:43:04.359 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:04.360 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:04.360 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_42 2022-04-10T01:43:04.360 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_42.status 2022-04-10T01:43:04.360 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:04.361 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:04.361 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:04.361 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:04.361 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2022-04-10T01:43:04.361 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:43:04.406 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:43:04.407 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:43:04.408 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:43:04.408 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:43:04.409 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:43:04.409 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:43:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 242 = '' 2022-04-10T01:43:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:43:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:43:05.211 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_42.status 2022-04-10T01:43:05.211 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:05.211 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_42 2022-04-10T01:43:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_42.status 2022-04-10T01:43:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2022-04-10T01:43:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:43:05.252 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:43:05.255 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=242 2022-04-10T01:43:05.255 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 242 = 242 2022-04-10T01:43:05.255 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:05.256 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror image_42 snap 2022-04-10T01:43:05.256 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:05.256 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:05.256 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_42 2022-04-10T01:43:05.256 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:43:05.257 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:43:05.257 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:05.257 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:05.257 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:05.258 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:05.258 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:43:05.258 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:43:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/image_42@snap 2022-04-10T01:43:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_42': 2022-04-10T01:43:06.298 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:43:06.298 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:43:06.298 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 2 2022-04-10T01:43:06.298 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 11336a0d5094 2022-04-10T01:43:06.299 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.11336a0d5094 2022-04-10T01:43:06.299 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:43:06.299 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:43:06.299 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:43:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:43:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:35:32 2022 2022-04-10T01:43:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:35:32 2022 2022-04-10T01:43:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:35:32 2022 2022-04-10T01:43:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:43:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:43:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:43:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: ceed605c-dff2-49f6-bdf7-f3ed1ce8dfcd 2022-04-10T01:43:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:43:06.303 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:06.303 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror image_42 snap 2022-04-10T01:43:06.303 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:06.304 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_42 2022-04-10T01:43:06.304 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:43:06.304 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:43:06.304 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_42.export 2022-04-10T01:43:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_42.export 2022-04-10T01:43:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_42.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_42.export 2022-04-10T01:43:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export image_42@snap /tmp/tmp.rgwz74WspX/cluster2-mirror-image_42.export 2022-04-10T01:43:06.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:43:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export image_42@snap /tmp/tmp.rgwz74WspX/cluster1-mirror-image_42.export 2022-04-10T01:43:06.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:43:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-image_42.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_42.export 2022-04-10T01:43:06.755 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_42.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_42.export 2022-04-10T01:43:06.792 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:06.793 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:43:06.793 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_43 2022-04-10T01:43:06.793 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror image_43 snap 2022-04-10T01:43:06.793 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:06.794 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:06.794 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_43 2022-04-10T01:43:06.794 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:43:06.794 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create image_43@snap --debug-rbd=20 --debug-journaler=20 2022-04-10T01:43:08.660 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror image_43 2022-04-10T01:43:08.661 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:08.661 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:08.661 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_43 2022-04-10T01:43:08.662 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:08.662 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:08.662 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:08.662 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:08.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror image_43 started 2022-04-10T01:43:08.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:08.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:08.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_43 2022-04-10T01:43:08.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:43:08.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:43:08.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:08.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:08.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:08.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:08.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:43:08.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:43:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror image_43 started 2022-04-10T01:43:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_43 2022-04-10T01:43:09.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:43:09.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:43:09.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:43:09.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:09.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:09.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:09.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:09.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_43 2022-04-10T01:43:09.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:43:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:43:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:43:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:43:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:43:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:43:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:43:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:43:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:43:09.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:43:09.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:43:09.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:43:09.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:43:09.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:43:09.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_43 2022-04-10T01:43:09.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:43:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:43:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:43:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:43:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:43:09.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:43:09.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:43:09.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:43:09.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:43:09.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:43:09.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:43:09.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:09.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:43:09.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:43:09.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:43:09.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:43:09.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:43:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:43:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:43:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:43:09.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:43:09.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:43:09.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:43:09.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:43:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:43:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:43:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:43:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:43:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:43:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:43:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:43:09.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:09.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:09.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:43:09.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:43:09.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:43:09.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:43:09.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:43:09.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_43 2022-04-10T01:43:09.809 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:43:09.809 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:43:09.809 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:43:09.810 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:43:09.810 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:43:09.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:43:09.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:43:09.816 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:09.816 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_43 2022-04-10T01:43:09.816 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:43:09.816 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:09.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:09.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_43 2022-04-10T01:43:09.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:43:09.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:09.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:09.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:09.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:43:09.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:43:09.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_43 2022-04-10T01:43:09.819 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:43:09.819 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:43:09.819 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:09.819 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_43 2022-04-10T01:43:09.819 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:43:09.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:09.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:09.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:09.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-image_43.status 2022-04-10T01:43:09.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-image_43.status 2022-04-10T01:43:09.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:43:09.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_43.status 2022-04-10T01:43:09.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-image_43.status 2022-04-10T01:43:09.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-image_43.status 2022-04-10T01:43:09.823 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:43:09.823 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_43.status 2022-04-10T01:43:09.823 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror image_43 2022-04-10T01:43:09.823 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:09.823 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:09.824 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_43 2022-04-10T01:43:09.824 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:43:09.824 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:09.824 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:09.825 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:09.825 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_43 2022-04-10T01:43:10.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 245 2022-04-10T01:43:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_43 /tmp/tmp.rgwz74WspX/cluster2-mirror-image_43.status 2022-04-10T01:43:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_43 2022-04-10T01:43:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_43.status 2022-04-10T01:43:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:43:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:10.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:10.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:10.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_43 --format xml 2022-04-10T01:43:10.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:43:10.719 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:43:10.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=245 2022-04-10T01:43:10.722 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:43:10.722 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:43:10.722 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:43:10.923 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_43.status 2022-04-10T01:43:10.924 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:10.924 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:10.924 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_43 2022-04-10T01:43:10.925 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_43.status 2022-04-10T01:43:10.925 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:10.925 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:10.925 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:10.925 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2022-04-10T01:43:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:43:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:43:10.973 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:43:10.974 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:43:10.974 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:43:10.974 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:43:10.975 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:43:10.976 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 245 = '' 2022-04-10T01:43:10.976 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:43:10.976 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:43:11.376 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_43.status 2022-04-10T01:43:11.377 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:11.377 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:11.377 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_43 2022-04-10T01:43:11.377 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_43.status 2022-04-10T01:43:11.378 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:11.378 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:11.378 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:11.378 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:11.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2022-04-10T01:43:11.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:43:11.423 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:43:11.424 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:43:11.425 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:43:11.426 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:43:11.426 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:43:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:43:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 245 = '' 2022-04-10T01:43:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:43:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:43:12.228 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_43.status 2022-04-10T01:43:12.228 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:12.229 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:12.229 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_43 2022-04-10T01:43:12.229 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_43.status 2022-04-10T01:43:12.229 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:12.229 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:12.230 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:12.230 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:12.230 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2022-04-10T01:43:12.230 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:43:12.270 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:43:12.273 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=245 2022-04-10T01:43:12.273 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 245 = 245 2022-04-10T01:43:12.274 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:12.274 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror image_43 snap 2022-04-10T01:43:12.275 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:12.275 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:12.275 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_43 2022-04-10T01:43:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:43:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:43:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:12.277 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:12.277 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:12.278 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:12.278 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:43:12.278 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:43:13.273 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/image_43@snap 2022-04-10T01:43:13.314 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_43': 2022-04-10T01:43:13.315 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:43:13.315 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:43:13.315 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 2 2022-04-10T01:43:13.315 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 12bafaa64559 2022-04-10T01:43:13.316 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.12bafaa64559 2022-04-10T01:43:13.316 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:43:13.316 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:43:13.316 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:43:13.317 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:43:13.317 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:35:43 2022 2022-04-10T01:43:13.317 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:35:43 2022 2022-04-10T01:43:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:35:43 2022 2022-04-10T01:43:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:43:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:43:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:43:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: bcd924f1-5832-47a7-92fc-1dbfdf17dc65 2022-04-10T01:43:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:43:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:13.321 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror image_43 snap 2022-04-10T01:43:13.321 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:13.321 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_43 2022-04-10T01:43:13.321 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:43:13.322 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:43:13.322 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_43.export 2022-04-10T01:43:13.322 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_43.export 2022-04-10T01:43:13.322 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_43.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_43.export 2022-04-10T01:43:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export image_43@snap /tmp/tmp.rgwz74WspX/cluster2-mirror-image_43.export 2022-04-10T01:43:13.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:43:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export image_43@snap /tmp/tmp.rgwz74WspX/cluster1-mirror-image_43.export 2022-04-10T01:43:13.738 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:43:13.741 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-image_43.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_43.export 2022-04-10T01:43:13.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_43.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_43.export 2022-04-10T01:43:13.842 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:13.842 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:43:13.842 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_44 2022-04-10T01:43:13.843 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror image_44 snap 2022-04-10T01:43:13.843 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:13.843 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:13.843 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_44 2022-04-10T01:43:13.844 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:43:13.844 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create image_44@snap --debug-rbd=20 --debug-journaler=20 2022-04-10T01:43:15.542 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror image_44 2022-04-10T01:43:15.542 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:15.542 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:15.543 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_44 2022-04-10T01:43:15.543 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:15.543 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:15.543 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:15.544 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:15.544 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror image_44 started 2022-04-10T01:43:15.544 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:15.544 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:15.544 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_44 2022-04-10T01:43:15.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:43:15.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:43:15.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:15.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:15.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:15.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:15.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:43:15.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:43:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror image_44 started 2022-04-10T01:43:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_44 2022-04-10T01:43:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:43:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:43:16.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:43:16.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:16.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:16.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:16.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:16.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_44 2022-04-10T01:43:16.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:43:16.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:43:16.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:43:16.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:43:16.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:43:16.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:43:16.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:43:16.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:43:16.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:16.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:43:16.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:43:16.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:43:16.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:43:16.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:43:16.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:43:16.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_44 2022-04-10T01:43:16.556 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:43:16.556 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:43:16.556 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:43:16.556 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:43:16.557 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:43:16.557 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:43:16.557 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:43:16.557 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:43:16.557 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:43:16.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:43:16.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:43:16.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:16.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:43:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:43:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:43:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:43:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:43:16.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:43:16.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:43:16.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:16.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:43:16.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:43:16.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:43:16.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:43:16.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:43:16.562 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:16.562 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:43:16.562 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:43:16.562 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:43:16.562 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:43:16.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:43:16.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:43:16.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:43:16.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:16.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:16.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:43:16.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:43:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:43:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:43:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:43:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_44 2022-04-10T01:43:16.681 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2022-04-10T01:43:16.681 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:43:16.681 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:43:16.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:43:16.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_44 2022-04-10T01:43:16.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:1 2022-04-10T01:43:16.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:43:16.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:43:16.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:1 cluster instance 2022-04-10T01:43:16.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:1 2022-04-10T01:43:16.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:43:16.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:43:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:43:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=1 2022-04-10T01:43:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 1 = cluster1:1 ']' 2022-04-10T01:43:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:43:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=1 2022-04-10T01:43:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:43:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:43:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:43:16.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:16.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:43:16.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:1 cluster1 2022-04-10T01:43:16.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:1 2022-04-10T01:43:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:43:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2022-04-10T01:43:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:1 2022-04-10T01:43:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:43:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:43:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:43:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 1 = cluster1:1 ']' 2022-04-10T01:43:16.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:43:16.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:43:16.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=1 2022-04-10T01:43:16.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=1 2022-04-10T01:43:16.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:43:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:43:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2022-04-10T01:43:16.700 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:43:16.700 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:43:16.700 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:43:16.700 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_44 2022-04-10T01:43:16.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2022-04-10T01:43:16.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:43:16.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:43:16.809 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:43:16.810 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_44 2022-04-10T01:43:16.810 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:2 2022-04-10T01:43:16.810 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:43:16.810 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:43:16.811 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:2 cluster instance 2022-04-10T01:43:16.811 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:2 2022-04-10T01:43:16.811 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:43:16.811 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:43:16.811 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:43:16.812 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=2 2022-04-10T01:43:16.812 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 2 = cluster1:2 ']' 2022-04-10T01:43:16.812 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:43:16.812 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:16.813 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=2 2022-04-10T01:43:16.813 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:43:16.813 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:43:16.813 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:43:16.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:16.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:43:16.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:2 cluster1 2022-04-10T01:43:16.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:2 2022-04-10T01:43:16.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:16.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:43:16.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2022-04-10T01:43:16.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:2 2022-04-10T01:43:16.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:43:16.816 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:43:16.816 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:16.816 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:43:16.816 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 2 = cluster1:2 ']' 2022-04-10T01:43:16.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:43:16.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:43:16.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=2 2022-04-10T01:43:16.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=2 2022-04-10T01:43:16.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:43:16.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:16.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:16.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:43:16.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2022-04-10T01:43:16.824 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:43:16.824 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:43:16.824 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:43:16.824 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_44 2022-04-10T01:43:16.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:43:16.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:43:16.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:43:16.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:43:16.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:43:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:43:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:43:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:16.945 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_44 2022-04-10T01:43:16.945 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:43:16.945 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:16.945 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:16.946 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_44 2022-04-10T01:43:16.946 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:43:16.946 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:16.946 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:43:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:43:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_44 2022-04-10T01:43:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:43:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:43:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_44 2022-04-10T01:43:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:16.949 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:16.949 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:16.949 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:16.949 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:43:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-image_44.status 2022-04-10T01:43:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-image_44.status 2022-04-10T01:43:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:43:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_44.status 2022-04-10T01:43:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-image_44.status 2022-04-10T01:43:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-image_44.status 2022-04-10T01:43:16.952 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:43:16.952 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_44.status 2022-04-10T01:43:16.952 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror image_44 2022-04-10T01:43:16.952 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:16.953 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:16.953 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_44 2022-04-10T01:43:16.953 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:43:16.953 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:16.953 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:16.954 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:16.954 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_44 2022-04-10T01:43:17.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 248 2022-04-10T01:43:17.557 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_44 /tmp/tmp.rgwz74WspX/cluster2-mirror-image_44.status 2022-04-10T01:43:17.557 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:17.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:17.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_44 2022-04-10T01:43:17.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_44.status 2022-04-10T01:43:17.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:43:17.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:17.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:17.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:17.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_44 --format xml 2022-04-10T01:43:17.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:43:17.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:43:17.601 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=248 2022-04-10T01:43:17.602 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:43:17.602 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:43:17.602 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:43:17.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_44.status 2022-04-10T01:43:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_44 2022-04-10T01:43:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_44.status 2022-04-10T01:43:17.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:17.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:17.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:17.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:17.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2022-04-10T01:43:17.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:43:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:43:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:43:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:43:17.849 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:43:17.849 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:43:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:43:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 248 = '' 2022-04-10T01:43:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:43:17.851 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:43:18.251 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_44.status 2022-04-10T01:43:18.251 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:18.252 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:18.252 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_44 2022-04-10T01:43:18.252 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_44.status 2022-04-10T01:43:18.252 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:18.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:18.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:18.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:18.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2022-04-10T01:43:18.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:43:18.301 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:43:18.302 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:43:18.302 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:43:18.302 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:43:18.302 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:43:18.302 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:43:18.303 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 248 = '' 2022-04-10T01:43:18.303 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:43:18.303 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:43:19.102 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_44.status 2022-04-10T01:43:19.103 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:19.103 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:19.103 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_44 2022-04-10T01:43:19.103 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_44.status 2022-04-10T01:43:19.104 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:19.104 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:19.104 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:19.104 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2022-04-10T01:43:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:43:19.143 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:43:19.147 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=248 2022-04-10T01:43:19.148 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 248 = 248 2022-04-10T01:43:19.148 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:19.148 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror image_44 snap 2022-04-10T01:43:19.148 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:19.148 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:19.149 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_44 2022-04-10T01:43:19.149 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:43:19.149 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:43:19.149 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:19.150 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:19.150 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:19.150 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:19.150 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:43:19.150 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:43:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/image_44@snap 2022-04-10T01:43:20.188 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_44': 2022-04-10T01:43:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:43:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:43:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 2 2022-04-10T01:43:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 11331526a961 2022-04-10T01:43:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.11331526a961 2022-04-10T01:43:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:43:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:43:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:43:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:43:20.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:35:58 2022 2022-04-10T01:43:20.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:35:58 2022 2022-04-10T01:43:20.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:35:58 2022 2022-04-10T01:43:20.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:43:20.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:43:20.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:43:20.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 06cfc5f2-3677-4f04-83bb-d75fb95a469d 2022-04-10T01:43:20.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:43:20.194 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:20.194 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror image_44 snap 2022-04-10T01:43:20.194 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:20.194 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_44 2022-04-10T01:43:20.195 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:43:20.195 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:43:20.195 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_44.export 2022-04-10T01:43:20.195 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_44.export 2022-04-10T01:43:20.195 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_44.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_44.export 2022-04-10T01:43:20.196 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export image_44@snap /tmp/tmp.rgwz74WspX/cluster2-mirror-image_44.export 2022-04-10T01:43:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:43:20.385 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export image_44@snap /tmp/tmp.rgwz74WspX/cluster1-mirror-image_44.export 2022-04-10T01:43:20.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:43:20.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-image_44.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_44.export 2022-04-10T01:43:20.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_44.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_44.export 2022-04-10T01:43:20.724 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:43:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_45 2022-04-10T01:43:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror image_45 snap 2022-04-10T01:43:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:20.726 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_45 2022-04-10T01:43:20.726 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:43:20.726 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create image_45@snap --debug-rbd=20 --debug-journaler=20 2022-04-10T01:43:22.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror image_45 2022-04-10T01:43:22.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:22.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:22.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_45 2022-04-10T01:43:22.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:22.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:22.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:22.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:22.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror image_45 started 2022-04-10T01:43:22.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:22.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:22.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_45 2022-04-10T01:43:22.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:43:22.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:43:22.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:22.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:22.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:22.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:22.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:43:22.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:43:23.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror image_45 started 2022-04-10T01:43:23.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:23.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:23.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_45 2022-04-10T01:43:23.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:43:23.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:43:23.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:43:23.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:23.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:23.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:23.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:23.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_45 2022-04-10T01:43:23.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:43:23.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:43:23.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:43:23.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:43:23.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:43:23.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:43:23.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:43:23.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:43:23.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:23.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:43:23.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:43:23.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:43:23.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:43:23.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:43:23.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:43:23.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_45 2022-04-10T01:43:23.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:43:23.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:43:23.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:43:23.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:43:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:43:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:43:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:43:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:43:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:43:23.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:43:23.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:43:23.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:23.562 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:43:23.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:43:23.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:43:23.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:43:23.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:23.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:43:23.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:43:23.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:43:23.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:23.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:43:23.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:43:23.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:43:23.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:43:23.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:43:23.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:23.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:43:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:43:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:43:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:43:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:43:23.567 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:43:23.567 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:43:23.567 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:23.567 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:23.567 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:43:23.568 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:43:23.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:43:23.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:43:23.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:43:23.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_45 2022-04-10T01:43:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2022-04-10T01:43:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:43:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:43:23.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:43:23.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_45 2022-04-10T01:43:23.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:1 2022-04-10T01:43:23.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:43:23.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:43:23.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:1 cluster instance 2022-04-10T01:43:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:1 2022-04-10T01:43:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:43:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:43:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:43:23.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=1 2022-04-10T01:43:23.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 1 = cluster1:1 ']' 2022-04-10T01:43:23.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:43:23.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:23.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=1 2022-04-10T01:43:23.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:43:23.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:43:23.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:43:23.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:23.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:43:23.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:1 cluster1 2022-04-10T01:43:23.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:1 2022-04-10T01:43:23.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:23.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:43:23.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2022-04-10T01:43:23.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:1 2022-04-10T01:43:23.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:43:23.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:43:23.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:23.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:43:23.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 1 = cluster1:1 ']' 2022-04-10T01:43:23.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:43:23.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:43:23.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=1 2022-04-10T01:43:23.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=1 2022-04-10T01:43:23.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:43:23.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:23.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:23.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:43:23.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2022-04-10T01:43:23.703 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:43:23.703 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:43:23.703 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:43:23.703 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_45 2022-04-10T01:43:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2022-04-10T01:43:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:43:23.810 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:43:23.813 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:43:23.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_45 2022-04-10T01:43:23.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:2 2022-04-10T01:43:23.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:43:23.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:43:23.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:2 cluster instance 2022-04-10T01:43:23.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:2 2022-04-10T01:43:23.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:43:23.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:43:23.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:43:23.816 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=2 2022-04-10T01:43:23.816 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 2 = cluster1:2 ']' 2022-04-10T01:43:23.816 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:43:23.816 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:23.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=2 2022-04-10T01:43:23.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:43:23.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:43:23.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:43:23.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:43:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:2 cluster1 2022-04-10T01:43:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:2 2022-04-10T01:43:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:43:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2022-04-10T01:43:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:2 2022-04-10T01:43:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:43:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:43:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:43:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 2 = cluster1:2 ']' 2022-04-10T01:43:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:43:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:43:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=2 2022-04-10T01:43:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=2 2022-04-10T01:43:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:43:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:43:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2022-04-10T01:43:23.828 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:43:23.828 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:43:23.829 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:43:23.829 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_45 2022-04-10T01:43:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2022-04-10T01:43:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:43:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:43:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:43:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_45 2022-04-10T01:43:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:3 2022-04-10T01:43:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:43:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:43:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:3 cluster instance 2022-04-10T01:43:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:3 2022-04-10T01:43:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:43:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:43:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:43:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=3 2022-04-10T01:43:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 3 = cluster1:3 ']' 2022-04-10T01:43:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:43:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=3 2022-04-10T01:43:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=3 2022-04-10T01:43:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:43:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:43:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:43:23.944 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:3 cluster1 2022-04-10T01:43:23.944 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:3 2022-04-10T01:43:23.944 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:23.944 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:43:23.945 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2022-04-10T01:43:23.945 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:3 2022-04-10T01:43:23.945 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:43:23.945 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:43:23.945 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:23.946 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=3 2022-04-10T01:43:23.946 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 3 = cluster1:3 ']' 2022-04-10T01:43:23.946 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:43:23.946 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:43:23.947 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=3 2022-04-10T01:43:23.947 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=3 2022-04-10T01:43:23.947 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:43:23.947 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:43:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2022-04-10T01:43:23.953 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:43:23.953 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:43:23.953 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:43:23.953 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_45 2022-04-10T01:43:24.068 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:43:24.069 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:43:24.069 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:43:24.069 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:43:24.070 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:43:24.074 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:43:24.074 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:43:24.075 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:24.075 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_45 2022-04-10T01:43:24.075 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:43:24.075 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:24.076 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:24.076 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_45 2022-04-10T01:43:24.076 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:43:24.076 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:43:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:43:24.078 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_45 2022-04-10T01:43:24.078 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:43:24.078 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:43:24.078 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:24.079 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_45 2022-04-10T01:43:24.079 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:24.079 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:24.079 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:24.080 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:24.080 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:43:24.080 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:24.080 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:24.081 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:24.081 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-image_45.status 2022-04-10T01:43:24.081 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-image_45.status 2022-04-10T01:43:24.081 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:43:24.081 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_45.status 2022-04-10T01:43:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-image_45.status 2022-04-10T01:43:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-image_45.status 2022-04-10T01:43:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:43:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_45.status 2022-04-10T01:43:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror image_45 2022-04-10T01:43:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_45 2022-04-10T01:43:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:43:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_45 2022-04-10T01:43:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 251 2022-04-10T01:43:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_45 /tmp/tmp.rgwz74WspX/cluster2-mirror-image_45.status 2022-04-10T01:43:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_45 2022-04-10T01:43:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_45.status 2022-04-10T01:43:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:43:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_45 --format xml 2022-04-10T01:43:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:43:24.601 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:43:24.603 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=251 2022-04-10T01:43:24.603 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:43:24.603 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:43:24.603 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:43:24.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_45.status 2022-04-10T01:43:24.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:24.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:24.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_45 2022-04-10T01:43:24.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_45.status 2022-04-10T01:43:24.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:24.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:24.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:24.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:24.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2022-04-10T01:43:24.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:43:24.849 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:43:24.850 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:43:24.851 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:43:24.851 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:43:24.851 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:43:24.852 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:43:24.852 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 251 = '' 2022-04-10T01:43:24.853 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:43:24.853 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:43:25.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_45.status 2022-04-10T01:43:25.254 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:25.254 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:25.254 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_45 2022-04-10T01:43:25.255 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_45.status 2022-04-10T01:43:25.255 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:25.255 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:25.255 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2022-04-10T01:43:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:43:25.301 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:43:25.301 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:43:25.302 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:43:25.303 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:43:25.303 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:43:25.304 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:43:25.304 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 251 = '' 2022-04-10T01:43:25.304 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:43:25.304 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:43:26.105 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_45.status 2022-04-10T01:43:26.106 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:26.106 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:26.106 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_45 2022-04-10T01:43:26.107 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_45.status 2022-04-10T01:43:26.107 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:26.107 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:26.107 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:26.107 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:26.108 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2022-04-10T01:43:26.108 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:43:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:43:26.148 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=251 2022-04-10T01:43:26.148 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 251 = 251 2022-04-10T01:43:26.149 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:26.149 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror image_45 snap 2022-04-10T01:43:26.149 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:26.149 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:26.149 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_45 2022-04-10T01:43:26.150 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:43:26.150 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:43:26.150 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:26.150 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:26.151 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:26.151 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:26.151 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:43:26.151 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:43:27.149 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/image_45@snap 2022-04-10T01:43:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_45': 2022-04-10T01:43:27.193 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:43:27.193 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:43:27.193 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 2 2022-04-10T01:43:27.193 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 10e4ccecefba 2022-04-10T01:43:27.194 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.10e4ccecefba 2022-04-10T01:43:27.194 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:43:27.194 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:43:27.194 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:43:27.195 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:43:27.195 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:36:06 2022 2022-04-10T01:43:27.195 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:36:06 2022 2022-04-10T01:43:27.195 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:36:06 2022 2022-04-10T01:43:27.195 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:43:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:43:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:43:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: eb158ebf-3a87-4b1d-a50e-a8e02e82428f 2022-04-10T01:43:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:43:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror image_45 snap 2022-04-10T01:43:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_45 2022-04-10T01:43:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:43:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:43:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_45.export 2022-04-10T01:43:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_45.export 2022-04-10T01:43:27.200 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_45.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_45.export 2022-04-10T01:43:27.200 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export image_45@snap /tmp/tmp.rgwz74WspX/cluster2-mirror-image_45.export 2022-04-10T01:43:27.376 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:43:27.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export image_45@snap /tmp/tmp.rgwz74WspX/cluster1-mirror-image_45.export 2022-04-10T01:43:27.615 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:43:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-image_45.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_45.export 2022-04-10T01:43:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_45.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_45.export 2022-04-10T01:43:27.719 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:27.720 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:43:27.720 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_46 2022-04-10T01:43:27.720 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror image_46 snap 2022-04-10T01:43:27.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:27.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:27.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_46 2022-04-10T01:43:27.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:43:27.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create image_46@snap --debug-rbd=20 --debug-journaler=20 2022-04-10T01:43:29.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror image_46 2022-04-10T01:43:29.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:29.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:29.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_46 2022-04-10T01:43:29.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:29.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:29.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:29.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:29.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror image_46 started 2022-04-10T01:43:29.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_46 2022-04-10T01:43:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:43:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:43:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:29.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:43:29.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:43:30.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror image_46 started 2022-04-10T01:43:30.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:30.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:30.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_46 2022-04-10T01:43:30.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:43:30.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:43:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:43:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:30.556 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_46 2022-04-10T01:43:30.556 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:43:30.556 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:43:30.556 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:43:30.556 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:43:30.557 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:43:30.557 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:43:30.557 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:43:30.557 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:43:30.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:30.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:43:30.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:43:30.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:43:30.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:43:30.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:43:30.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:43:30.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_46 2022-04-10T01:43:30.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:43:30.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:43:30.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:43:30.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:43:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:43:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:43:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:43:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:43:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:43:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:43:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:43:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:43:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:43:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:43:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:43:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:43:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:43:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:43:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:43:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:43:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:43:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:43:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:43:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:43:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:43:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:43:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:43:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:43:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:43:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:43:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:43:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:43:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:43:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:43:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:43:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_46 2022-04-10T01:43:30.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2022-04-10T01:43:30.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:43:30.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:43:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:43:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_46 2022-04-10T01:43:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:1 2022-04-10T01:43:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:43:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:43:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:1 cluster instance 2022-04-10T01:43:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:1 2022-04-10T01:43:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:43:30.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:43:30.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:43:30.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=1 2022-04-10T01:43:30.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 1 = cluster1:1 ']' 2022-04-10T01:43:30.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:43:30.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:30.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=1 2022-04-10T01:43:30.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:43:30.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:43:30.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:43:30.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:30.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:43:30.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:1 cluster1 2022-04-10T01:43:30.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:1 2022-04-10T01:43:30.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:30.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:43:30.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2022-04-10T01:43:30.698 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:1 2022-04-10T01:43:30.698 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:43:30.698 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:43:30.698 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:30.699 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:43:30.699 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 1 = cluster1:1 ']' 2022-04-10T01:43:30.699 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:43:30.699 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:43:30.699 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=1 2022-04-10T01:43:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=1 2022-04-10T01:43:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:43:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:43:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2022-04-10T01:43:30.707 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:43:30.707 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:43:30.707 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:43:30.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_46 2022-04-10T01:43:30.812 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2022-04-10T01:43:30.812 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:43:30.812 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:43:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:43:30.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_46 2022-04-10T01:43:30.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:2 2022-04-10T01:43:30.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:43:30.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:43:30.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:2 cluster instance 2022-04-10T01:43:30.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:2 2022-04-10T01:43:30.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:43:30.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:43:30.819 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:43:30.819 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=2 2022-04-10T01:43:30.819 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 2 = cluster1:2 ']' 2022-04-10T01:43:30.819 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:43:30.819 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:30.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=2 2022-04-10T01:43:30.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:43:30.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:43:30.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:43:30.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:30.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:43:30.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:2 cluster1 2022-04-10T01:43:30.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:2 2022-04-10T01:43:30.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:30.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:43:30.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2022-04-10T01:43:30.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:2 2022-04-10T01:43:30.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:43:30.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:43:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:43:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 2 = cluster1:2 ']' 2022-04-10T01:43:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:43:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:43:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=2 2022-04-10T01:43:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=2 2022-04-10T01:43:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:43:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:43:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2022-04-10T01:43:30.830 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:43:30.831 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:43:30.831 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:43:30.831 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_46 2022-04-10T01:43:30.947 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:43:30.948 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:43:30.948 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:43:30.948 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:43:30.949 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:43:30.952 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:43:30.952 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:43:30.952 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:30.953 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_46 2022-04-10T01:43:30.953 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:43:30.953 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:30.953 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:30.954 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_46 2022-04-10T01:43:30.954 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:43:30.954 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:30.954 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:30.954 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:30.955 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:43:30.955 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:43:30.955 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_46 2022-04-10T01:43:30.955 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:43:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:43:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:30.957 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_46 2022-04-10T01:43:30.958 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:30.958 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:30.958 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:30.958 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:30.959 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:43:30.959 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:30.959 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:30.959 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:30.959 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-image_46.status 2022-04-10T01:43:30.960 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-image_46.status 2022-04-10T01:43:30.960 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:43:30.960 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_46.status 2022-04-10T01:43:30.960 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-image_46.status 2022-04-10T01:43:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-image_46.status 2022-04-10T01:43:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:43:30.962 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_46.status 2022-04-10T01:43:30.962 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror image_46 2022-04-10T01:43:30.963 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:30.963 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:30.963 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_46 2022-04-10T01:43:30.963 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:43:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_46 2022-04-10T01:43:31.568 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 254 2022-04-10T01:43:31.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_46 /tmp/tmp.rgwz74WspX/cluster2-mirror-image_46.status 2022-04-10T01:43:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_46 2022-04-10T01:43:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_46.status 2022-04-10T01:43:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:43:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_46 --format xml 2022-04-10T01:43:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:43:31.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:43:31.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=254 2022-04-10T01:43:31.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:43:31.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:43:31.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:43:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_46.status 2022-04-10T01:43:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_46 2022-04-10T01:43:31.825 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_46.status 2022-04-10T01:43:31.825 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:31.825 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:31.825 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:31.826 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:31.826 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2022-04-10T01:43:31.826 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:43:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:43:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:43:31.869 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:43:31.869 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:43:31.870 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:43:31.871 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:43:31.871 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 254 = '' 2022-04-10T01:43:31.871 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:43:31.871 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:43:32.272 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_46.status 2022-04-10T01:43:32.272 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:32.272 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_46 2022-04-10T01:43:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_46.status 2022-04-10T01:43:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:32.274 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:32.274 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2022-04-10T01:43:32.274 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:43:32.319 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:43:32.319 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:43:32.321 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:43:32.321 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:43:32.321 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:43:32.322 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:43:32.322 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 254 = '' 2022-04-10T01:43:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:43:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:43:33.123 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_46.status 2022-04-10T01:43:33.124 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:33.124 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:33.124 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_46 2022-04-10T01:43:33.124 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_46.status 2022-04-10T01:43:33.124 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:33.125 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:33.125 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:33.125 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:33.125 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2022-04-10T01:43:33.126 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:43:33.166 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:43:33.168 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=254 2022-04-10T01:43:33.168 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 254 = 254 2022-04-10T01:43:33.168 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:33.169 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror image_46 snap 2022-04-10T01:43:33.169 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:33.169 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:33.169 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_46 2022-04-10T01:43:33.170 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:43:33.170 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:43:33.170 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:33.170 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:33.170 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:33.171 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:33.171 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:43:33.171 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:43:34.169 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/image_46@snap 2022-04-10T01:43:34.212 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_46': 2022-04-10T01:43:34.213 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:43:34.213 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:43:34.213 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 2 2022-04-10T01:43:34.213 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 1133a4e980a6 2022-04-10T01:43:34.214 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.1133a4e980a6 2022-04-10T01:43:34.214 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:43:34.214 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:43:34.214 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:43:34.215 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:43:34.215 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:36:19 2022 2022-04-10T01:43:34.215 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:36:19 2022 2022-04-10T01:43:34.215 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:36:19 2022 2022-04-10T01:43:34.215 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:43:34.216 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:43:34.216 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:43:34.216 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: c83a7972-17a4-4b3c-b33c-85a8de6039e5 2022-04-10T01:43:34.216 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:43:34.218 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:34.218 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror image_46 snap 2022-04-10T01:43:34.218 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:34.218 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_46 2022-04-10T01:43:34.218 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:43:34.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:43:34.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_46.export 2022-04-10T01:43:34.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_46.export 2022-04-10T01:43:34.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_46.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_46.export 2022-04-10T01:43:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export image_46@snap /tmp/tmp.rgwz74WspX/cluster2-mirror-image_46.export 2022-04-10T01:43:34.396 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:43:34.399 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export image_46@snap /tmp/tmp.rgwz74WspX/cluster1-mirror-image_46.export 2022-04-10T01:43:34.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:43:34.616 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-image_46.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_46.export 2022-04-10T01:43:34.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_46.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_46.export 2022-04-10T01:43:34.717 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:34.717 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:43:34.718 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_47 2022-04-10T01:43:34.718 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror image_47 snap 2022-04-10T01:43:34.718 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:34.718 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_47 2022-04-10T01:43:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:43:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create image_47@snap --debug-rbd=20 --debug-journaler=20 2022-04-10T01:43:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror image_47 2022-04-10T01:43:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_47 2022-04-10T01:43:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror image_47 started 2022-04-10T01:43:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_47 2022-04-10T01:43:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:43:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:43:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:43:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:43:36.871 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5858342705856293) > weight (0.25) 2022-04-10T01:43:36.872 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2022-04-10T01:43:36.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2022-04-10T01:43:36.873 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2022-04-10T01:43:36.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi043.stderr:2022-04-10T01:43:36.871+0000 7f6e8983a700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 57949) UID: 1000 2022-04-10T01:43:37.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi043.stderr:daemon-helper: command failed with exit status 1 2022-04-10T01:43:37.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror image_47 started 2022-04-10T01:43:37.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:37.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:37.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_47 2022-04-10T01:43:37.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:43:37.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:43:37.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:43:37.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:37.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:37.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:37.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:37.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_47 2022-04-10T01:43:37.583 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:43:37.583 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:43:37.583 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:43:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:43:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:43:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:43:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:43:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:43:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:43:37.590 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:43:37.590 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:43:37.590 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:43:37.590 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:43:37.591 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:43:37.591 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_47 2022-04-10T01:43:37.591 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:43:37.591 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:43:37.591 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:43:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:43:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:43:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:43:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:43:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:43:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:43:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:43:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:43:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:37.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:43:37.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:43:37.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:43:37.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:43:37.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:37.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:43:37.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:43:37.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:43:37.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:37.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:43:37.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:43:37.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:43:37.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:43:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:43:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:43:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:43:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:43:37.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:43:37.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:43:37.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:43:37.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:43:37.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:37.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:37.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:43:37.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:43:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:43:37.603 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:43:37.603 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:43:37.603 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_47 2022-04-10T01:43:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2022-04-10T01:43:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:43:37.717 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:43:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:43:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_47 2022-04-10T01:43:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:1 2022-04-10T01:43:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:43:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:43:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:1 cluster instance 2022-04-10T01:43:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:1 2022-04-10T01:43:37.723 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:43:37.723 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:43:37.723 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:43:37.723 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=1 2022-04-10T01:43:37.724 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 1 = cluster1:1 ']' 2022-04-10T01:43:37.724 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:43:37.724 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:37.724 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=1 2022-04-10T01:43:37.724 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:43:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:43:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:43:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:43:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:1 cluster1 2022-04-10T01:43:37.726 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:1 2022-04-10T01:43:37.726 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:37.726 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:43:37.726 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2022-04-10T01:43:37.727 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:1 2022-04-10T01:43:37.727 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:43:37.727 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:43:37.727 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:37.727 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:43:37.728 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 1 = cluster1:1 ']' 2022-04-10T01:43:37.728 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:43:37.728 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:43:37.728 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=1 2022-04-10T01:43:37.729 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=1 2022-04-10T01:43:37.729 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:43:37.729 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:37.729 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:37.729 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:43:37.730 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2022-04-10T01:43:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:43:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:43:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:43:37.736 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_47 2022-04-10T01:43:37.819 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2022-04-10T01:43:37.823 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:43:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_47 2022-04-10T01:43:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:2 2022-04-10T01:43:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:43:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:43:37.825 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:2 cluster instance 2022-04-10T01:43:37.825 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:2 2022-04-10T01:43:37.825 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:43:37.825 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:43:37.825 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:43:37.826 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=2 2022-04-10T01:43:37.826 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 2 = cluster1:2 ']' 2022-04-10T01:43:37.826 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:43:37.826 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:37.826 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=2 2022-04-10T01:43:37.827 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:43:37.827 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:43:37.827 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:43:37.827 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:37.827 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:43:37.828 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:2 cluster1 2022-04-10T01:43:37.828 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:2 2022-04-10T01:43:37.828 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:37.828 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:43:37.828 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2022-04-10T01:43:37.829 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:2 2022-04-10T01:43:37.829 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:43:37.829 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:43:37.829 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:37.830 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:43:37.830 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 2 = cluster1:2 ']' 2022-04-10T01:43:37.830 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:43:37.830 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:43:37.830 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=2 2022-04-10T01:43:37.831 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=2 2022-04-10T01:43:37.831 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:43:37.831 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:37.831 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:37.831 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:43:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2022-04-10T01:43:37.838 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:43:37.838 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:43:37.838 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:43:37.838 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_47 2022-04-10T01:43:37.944 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2022-04-10T01:43:37.944 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:43:37.944 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:43:37.948 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:43:37.948 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_47 2022-04-10T01:43:37.948 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:3 2022-04-10T01:43:37.949 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:43:37.949 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:43:37.949 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:3 cluster instance 2022-04-10T01:43:37.949 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:3 2022-04-10T01:43:37.949 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:43:37.950 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:43:37.950 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:43:37.950 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=3 2022-04-10T01:43:37.950 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 3 = cluster1:3 ']' 2022-04-10T01:43:37.951 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:43:37.951 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:37.951 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=3 2022-04-10T01:43:37.951 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=3 2022-04-10T01:43:37.951 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:43:37.952 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:43:37.952 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:37.952 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:43:37.952 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:3 cluster1 2022-04-10T01:43:37.952 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:3 2022-04-10T01:43:37.953 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:37.953 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:43:37.953 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2022-04-10T01:43:37.953 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:3 2022-04-10T01:43:37.953 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:43:37.954 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:43:37.954 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:37.954 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=3 2022-04-10T01:43:37.954 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 3 = cluster1:3 ']' 2022-04-10T01:43:37.954 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:43:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:43:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=3 2022-04-10T01:43:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=3 2022-04-10T01:43:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:43:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:37.956 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:37.956 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:43:37.956 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2022-04-10T01:43:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:43:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:43:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:43:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_47 2022-04-10T01:43:38.079 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:43:38.080 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:43:38.080 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:43:38.080 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:43:38.081 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:43:38.084 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:43:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:43:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_47 2022-04-10T01:43:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:43:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_47 2022-04-10T01:43:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:43:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:38.087 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:38.087 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:38.087 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:43:38.087 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:43:38.087 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_47 2022-04-10T01:43:38.088 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:43:38.088 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:43:38.088 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:38.088 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_47 2022-04-10T01:43:38.088 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:43:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-image_47.status 2022-04-10T01:43:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-image_47.status 2022-04-10T01:43:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:43:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_47.status 2022-04-10T01:43:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-image_47.status 2022-04-10T01:43:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-image_47.status 2022-04-10T01:43:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:43:38.093 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_47.status 2022-04-10T01:43:38.093 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror image_47 2022-04-10T01:43:38.093 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:38.094 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:38.094 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_47 2022-04-10T01:43:38.094 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:43:38.094 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_47 2022-04-10T01:43:38.584 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 257 2022-04-10T01:43:38.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_47 /tmp/tmp.rgwz74WspX/cluster2-mirror-image_47.status 2022-04-10T01:43:38.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:38.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:38.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_47 2022-04-10T01:43:38.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_47.status 2022-04-10T01:43:38.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:43:38.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:38.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:38.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:38.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_47 --format xml 2022-04-10T01:43:38.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:43:38.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:43:38.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=257 2022-04-10T01:43:38.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:43:38.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:43:38.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:43:38.840 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_47.status 2022-04-10T01:43:38.840 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:38.841 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:38.841 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_47 2022-04-10T01:43:38.841 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_47.status 2022-04-10T01:43:38.841 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:38.841 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2022-04-10T01:43:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:43:38.886 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:43:38.886 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:43:38.887 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:43:38.887 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:43:38.888 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:43:38.888 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:43:38.889 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 257 = '' 2022-04-10T01:43:38.889 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:43:38.889 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:43:39.290 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_47.status 2022-04-10T01:43:39.290 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:39.290 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:39.291 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_47 2022-04-10T01:43:39.291 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_47.status 2022-04-10T01:43:39.291 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:39.291 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:39.291 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:39.292 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:39.292 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2022-04-10T01:43:39.292 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:43:39.338 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:43:39.339 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:43:39.340 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:43:39.340 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:43:39.340 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:43:39.341 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:43:39.341 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 257 = '' 2022-04-10T01:43:39.342 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:43:39.342 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:43:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_47.status 2022-04-10T01:43:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:40.144 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_47 2022-04-10T01:43:40.144 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_47.status 2022-04-10T01:43:40.144 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:40.144 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:40.144 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2022-04-10T01:43:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:43:40.186 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:43:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=257 2022-04-10T01:43:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 257 = 257 2022-04-10T01:43:40.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:40.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror image_47 snap 2022-04-10T01:43:40.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:40.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:40.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_47 2022-04-10T01:43:40.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:43:40.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:43:40.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:40.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:40.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:40.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:40.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:43:40.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:43:41.188 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/image_47@snap 2022-04-10T01:43:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_47': 2022-04-10T01:43:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:43:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:43:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 2 2022-04-10T01:43:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 118dc8ced1e8 2022-04-10T01:43:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.118dc8ced1e8 2022-04-10T01:43:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:43:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:43:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:43:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:43:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:36:34 2022 2022-04-10T01:43:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:36:34 2022 2022-04-10T01:43:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:36:34 2022 2022-04-10T01:43:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:43:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:43:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:43:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: a7fe8d55-51fd-4578-8dbb-89aefafe026d 2022-04-10T01:43:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:43:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror image_47 snap 2022-04-10T01:43:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_47 2022-04-10T01:43:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:43:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:43:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_47.export 2022-04-10T01:43:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_47.export 2022-04-10T01:43:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_47.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_47.export 2022-04-10T01:43:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export image_47@snap /tmp/tmp.rgwz74WspX/cluster2-mirror-image_47.export 2022-04-10T01:43:41.412 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:43:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export image_47@snap /tmp/tmp.rgwz74WspX/cluster1-mirror-image_47.export 2022-04-10T01:43:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:43:41.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-image_47.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_47.export 2022-04-10T01:43:41.727 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_47.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_47.export 2022-04-10T01:43:41.766 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:41.766 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:43:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_48 2022-04-10T01:43:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror image_48 snap 2022-04-10T01:43:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:41.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_48 2022-04-10T01:43:41.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:43:41.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create image_48@snap --debug-rbd=20 --debug-journaler=20 2022-04-10T01:43:42.874 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2022-04-10T01:43:42.875 INFO:teuthology.orchestra.run:waiting for 600 2022-04-10T01:43:42.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-04-10T01:43:42.876 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2022-04-10T01:43:42.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2022-04-10T01:43:42.877 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2022-04-10T01:43:42.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2022-04-10T01:43:42.881 INFO:tasks.rbd_mirror_thrash:waiting for 74 secs before thrashing 2022-04-10T01:43:43.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror image_48 2022-04-10T01:43:43.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:43.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:43.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_48 2022-04-10T01:43:43.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:43.600 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:43.600 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:43.600 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:43.600 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror image_48 started 2022-04-10T01:43:43.600 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_48 2022-04-10T01:43:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:43:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:43:43.602 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:43.602 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:43.602 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:43.602 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:43.602 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:43:43.603 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:43:44.382 INFO:tasks.ceph.cluster1.osd.2.smithi005.stderr:2022-04-10T01:43:44.380+0000 7f0c60f96700 -1 reset not still connected to 0x564c1a578ea0 2022-04-10T01:43:44.601 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror image_48 started 2022-04-10T01:43:44.601 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:44.601 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:44.601 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_48 2022-04-10T01:43:44.602 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:43:44.602 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:43:44.602 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:43:44.602 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:44.603 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:44.603 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:44.603 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:44.603 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_48 2022-04-10T01:43:44.603 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:43:44.604 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:43:44.604 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:43:44.604 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:43:44.604 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:43:44.605 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:43:44.605 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:43:44.605 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:43:44.605 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:44.605 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:43:44.606 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:43:44.606 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:43:44.606 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:43:44.610 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:43:44.610 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:43:44.610 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_48 2022-04-10T01:43:44.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:43:44.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:43:44.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:43:44.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:43:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:43:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:43:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:43:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:43:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:43:44.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:43:44.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:43:44.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:44.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:43:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:43:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:43:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:43:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:44.615 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:43:44.615 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:43:44.615 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:43:44.615 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:44.615 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:43:44.616 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:43:44.616 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:43:44.616 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:43:44.616 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:43:44.616 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:43:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:43:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:43:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:43:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:43:44.618 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:43:44.618 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:43:44.618 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:44.618 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:43:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:43:44.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:43:44.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:43:44.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:43:44.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_48 2022-04-10T01:43:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2022-04-10T01:43:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:43:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:43:44.741 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:43:44.741 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_48 2022-04-10T01:43:44.742 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:1 2022-04-10T01:43:44.742 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:43:44.742 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:43:44.742 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:1 cluster instance 2022-04-10T01:43:44.743 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:1 2022-04-10T01:43:44.743 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:43:44.743 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:43:44.743 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:43:44.744 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=1 2022-04-10T01:43:44.744 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 1 = cluster1:1 ']' 2022-04-10T01:43:44.744 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:43:44.744 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:44.744 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=1 2022-04-10T01:43:44.745 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:43:44.745 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:43:44.745 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:43:44.745 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:44.746 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:43:44.746 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:1 cluster1 2022-04-10T01:43:44.746 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:1 2022-04-10T01:43:44.746 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:44.746 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:43:44.747 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2022-04-10T01:43:44.747 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:1 2022-04-10T01:43:44.747 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:43:44.747 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:43:44.748 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:44.748 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:43:44.748 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 1 = cluster1:1 ']' 2022-04-10T01:43:44.748 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:43:44.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:43:44.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=1 2022-04-10T01:43:44.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=1 2022-04-10T01:43:44.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:43:44.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:44.750 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:44.750 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:43:44.750 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2022-04-10T01:43:44.755 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:43:44.756 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:43:44.756 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:43:44.756 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_48 2022-04-10T01:43:44.849 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2022-04-10T01:43:44.850 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:43:44.850 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:43:44.853 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:43:44.854 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_48 2022-04-10T01:43:44.854 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:2 2022-04-10T01:43:44.854 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:43:44.854 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:43:44.855 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:2 cluster instance 2022-04-10T01:43:44.855 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:2 2022-04-10T01:43:44.855 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:43:44.855 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:43:44.856 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:43:44.856 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=2 2022-04-10T01:43:44.856 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 2 = cluster1:2 ']' 2022-04-10T01:43:44.856 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:43:44.856 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:44.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=2 2022-04-10T01:43:44.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:43:44.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:43:44.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:43:44.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:44.858 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:43:44.858 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:2 cluster1 2022-04-10T01:43:44.858 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:2 2022-04-10T01:43:44.859 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:44.859 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:43:44.859 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2022-04-10T01:43:44.859 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:2 2022-04-10T01:43:44.860 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:43:44.860 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:43:44.860 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:44.860 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:43:44.861 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 2 = cluster1:2 ']' 2022-04-10T01:43:44.861 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:43:44.861 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:43:44.861 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=2 2022-04-10T01:43:44.862 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=2 2022-04-10T01:43:44.862 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:43:44.862 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:44.862 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:44.862 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:43:44.863 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2022-04-10T01:43:44.867 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:43:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:43:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:43:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_48 2022-04-10T01:43:44.974 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2022-04-10T01:43:44.974 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:43:44.974 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:43:44.979 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:43:44.979 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_48 2022-04-10T01:43:44.979 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:3 2022-04-10T01:43:44.979 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:43:44.980 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:43:44.980 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:3 cluster instance 2022-04-10T01:43:44.980 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:3 2022-04-10T01:43:44.980 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:43:44.980 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:43:44.981 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:43:44.981 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=3 2022-04-10T01:43:44.981 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 3 = cluster1:3 ']' 2022-04-10T01:43:44.981 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:43:44.982 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:44.982 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=3 2022-04-10T01:43:44.982 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=3 2022-04-10T01:43:44.982 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:43:44.982 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:43:44.983 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:44.983 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:43:44.983 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:3 cluster1 2022-04-10T01:43:44.983 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:3 2022-04-10T01:43:44.983 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:44.984 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:43:44.984 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2022-04-10T01:43:44.984 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:3 2022-04-10T01:43:44.984 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:43:44.984 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:43:44.985 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:44.985 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=3 2022-04-10T01:43:44.985 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 3 = cluster1:3 ']' 2022-04-10T01:43:44.985 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:43:44.986 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:43:44.986 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=3 2022-04-10T01:43:44.986 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=3 2022-04-10T01:43:44.986 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:43:44.986 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:44.987 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:44.987 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:43:44.987 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2022-04-10T01:43:44.993 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:43:44.993 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:43:44.994 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:43:44.994 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_48 2022-04-10T01:43:45.108 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:43:45.109 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:43:45.109 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:43:45.110 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:43:45.110 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:43:45.113 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:43:45.113 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:43:45.114 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:45.114 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_48 2022-04-10T01:43:45.114 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:43:45.114 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_48 2022-04-10T01:43:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:43:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:45.116 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:45.116 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:43:45.116 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:43:45.116 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_48 2022-04-10T01:43:45.116 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:43:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:43:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_48 2022-04-10T01:43:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:45.118 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:45.118 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:45.118 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:45.118 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:43:45.118 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:45.119 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:45.119 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:45.119 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-image_48.status 2022-04-10T01:43:45.119 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-image_48.status 2022-04-10T01:43:45.120 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:43:45.120 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_48.status 2022-04-10T01:43:45.120 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-image_48.status 2022-04-10T01:43:45.120 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-image_48.status 2022-04-10T01:43:45.120 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:43:45.122 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_48.status 2022-04-10T01:43:45.122 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror image_48 2022-04-10T01:43:45.122 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:45.122 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:45.123 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_48 2022-04-10T01:43:45.123 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:43:45.123 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:45.123 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:45.123 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:45.124 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_48 2022-04-10T01:43:45.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 260 2022-04-10T01:43:45.607 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_48 /tmp/tmp.rgwz74WspX/cluster2-mirror-image_48.status 2022-04-10T01:43:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_48 2022-04-10T01:43:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_48.status 2022-04-10T01:43:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:43:45.609 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:45.609 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:45.609 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:45.609 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_48 --format xml 2022-04-10T01:43:45.609 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:43:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:43:45.649 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=260 2022-04-10T01:43:45.649 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:43:45.650 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:43:45.650 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:43:45.851 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_48.status 2022-04-10T01:43:45.851 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:45.852 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:45.852 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_48 2022-04-10T01:43:45.852 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_48.status 2022-04-10T01:43:45.852 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:45.853 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:45.853 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:45.853 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:45.853 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2022-04-10T01:43:45.854 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:43:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:43:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:43:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:43:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:43:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:43:45.902 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:43:45.902 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 260 = '' 2022-04-10T01:43:45.902 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:43:45.903 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:43:46.303 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_48.status 2022-04-10T01:43:46.303 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:46.304 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:46.304 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_48 2022-04-10T01:43:46.304 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_48.status 2022-04-10T01:43:46.304 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:46.304 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2022-04-10T01:43:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:43:46.348 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:43:46.349 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:43:46.350 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:43:46.350 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:43:46.350 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:43:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:43:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 260 = '' 2022-04-10T01:43:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:43:46.352 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:43:47.152 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_48.status 2022-04-10T01:43:47.153 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:47.153 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:47.153 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_48 2022-04-10T01:43:47.153 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_48.status 2022-04-10T01:43:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2022-04-10T01:43:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:43:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:43:47.197 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=260 2022-04-10T01:43:47.197 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 260 = 260 2022-04-10T01:43:47.197 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:47.197 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror image_48 snap 2022-04-10T01:43:47.197 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_48 2022-04-10T01:43:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:43:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:43:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:43:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:43:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/image_48@snap 2022-04-10T01:43:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_48': 2022-04-10T01:43:48.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:43:48.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:43:48.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 2 2022-04-10T01:43:48.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 12badde34c2a 2022-04-10T01:43:48.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.12badde34c2a 2022-04-10T01:43:48.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:43:48.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:43:48.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:43:48.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:43:48.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:36:47 2022 2022-04-10T01:43:48.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:36:47 2022 2022-04-10T01:43:48.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:36:47 2022 2022-04-10T01:43:48.242 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:43:48.242 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:43:48.242 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:43:48.242 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: dc4e02d6-0aa7-4947-95ef-025b13fdc64d 2022-04-10T01:43:48.243 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:43:48.244 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:48.244 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror image_48 snap 2022-04-10T01:43:48.244 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:48.245 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_48 2022-04-10T01:43:48.245 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:43:48.245 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:43:48.245 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_48.export 2022-04-10T01:43:48.246 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_48.export 2022-04-10T01:43:48.246 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_48.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_48.export 2022-04-10T01:43:48.246 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export image_48@snap /tmp/tmp.rgwz74WspX/cluster2-mirror-image_48.export 2022-04-10T01:43:48.429 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:43:48.432 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export image_48@snap /tmp/tmp.rgwz74WspX/cluster1-mirror-image_48.export 2022-04-10T01:43:48.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:43:48.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-image_48.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_48.export 2022-04-10T01:43:48.736 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_48.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_48.export 2022-04-10T01:43:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:43:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_49 2022-04-10T01:43:48.775 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror image_49 snap 2022-04-10T01:43:48.775 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:48.775 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:48.775 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_49 2022-04-10T01:43:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:43:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create image_49@snap --debug-rbd=20 --debug-journaler=20 2022-04-10T01:43:50.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror image_49 2022-04-10T01:43:50.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:50.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:50.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_49 2022-04-10T01:43:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror image_49 started 2022-04-10T01:43:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_49 2022-04-10T01:43:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:43:50.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:43:50.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:50.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:50.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:43:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:43:51.226 INFO:tasks.ceph.cluster2.osd.2.smithi043.stderr:2022-04-10T01:43:51.225+0000 7f8be0453700 -1 reset not still connected to 0x55f2b63976c0 2022-04-10T01:43:51.227 INFO:tasks.ceph.cluster2.osd.2.smithi043.stderr:2022-04-10T01:43:51.225+0000 7f8be0453700 -1 reset not still connected to 0x55f2b7f1a4e0 2022-04-10T01:43:51.227 INFO:tasks.ceph.cluster2.osd.2.smithi043.stderr:2022-04-10T01:43:51.225+0000 7f8be0453700 -1 reset not still connected to 0x55f2bcde2340 2022-04-10T01:43:51.227 INFO:tasks.ceph.cluster2.osd.2.smithi043.stderr:2022-04-10T01:43:51.225+0000 7f8be0453700 -1 reset not still connected to 0x55f2bcde2f70 2022-04-10T01:43:51.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror image_49 started 2022-04-10T01:43:51.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:51.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:51.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_49 2022-04-10T01:43:51.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:43:51.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:43:51.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:43:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_49 2022-04-10T01:43:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:43:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:43:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:43:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:43:51.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:43:51.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:43:51.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:43:51.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:43:51.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:51.556 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:43:51.556 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:43:51.556 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:43:51.556 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:43:51.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:43:51.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:43:51.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_49 2022-04-10T01:43:51.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:43:51.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:43:51.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:43:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:43:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:43:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:43:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:43:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:43:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:43:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:43:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:43:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:43:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:43:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:43:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:43:51.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:51.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:43:51.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:43:51.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:43:51.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:51.566 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:43:51.566 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:43:51.566 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:43:51.566 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:43:51.567 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:43:51.567 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:51.567 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:43:51.567 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:43:51.567 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:43:51.568 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:43:51.568 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:43:51.568 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:43:51.568 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:43:51.568 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:51.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:51.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:43:51.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:43:51.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:43:51.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:43:51.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:43:51.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_49 2022-04-10T01:43:51.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2022-04-10T01:43:51.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:43:51.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:43:51.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:43:51.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_49 2022-04-10T01:43:51.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:1 2022-04-10T01:43:51.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:43:51.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:43:51.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:1 cluster instance 2022-04-10T01:43:51.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:1 2022-04-10T01:43:51.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:43:51.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:43:51.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:43:51.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=1 2022-04-10T01:43:51.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 1 = cluster1:1 ']' 2022-04-10T01:43:51.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:43:51.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:51.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=1 2022-04-10T01:43:51.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:43:51.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:43:51.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:43:51.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:51.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:43:51.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:1 cluster1 2022-04-10T01:43:51.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:1 2022-04-10T01:43:51.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:51.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:43:51.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2022-04-10T01:43:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:1 2022-04-10T01:43:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:43:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:43:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:51.699 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:43:51.699 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 1 = cluster1:1 ']' 2022-04-10T01:43:51.699 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:43:51.699 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:43:51.700 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=1 2022-04-10T01:43:51.700 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=1 2022-04-10T01:43:51.700 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:43:51.700 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:43:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2022-04-10T01:43:51.706 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:43:51.706 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:43:51.707 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:43:51.707 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_49 2022-04-10T01:43:51.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2022-04-10T01:43:51.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:43:51.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:43:51.812 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:43:51.812 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_49 2022-04-10T01:43:51.813 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:2 2022-04-10T01:43:51.813 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:43:51.813 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:43:51.813 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:2 cluster instance 2022-04-10T01:43:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:2 2022-04-10T01:43:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:43:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:43:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:43:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=2 2022-04-10T01:43:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 2 = cluster1:2 ']' 2022-04-10T01:43:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:43:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=2 2022-04-10T01:43:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:43:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:43:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:43:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:51.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:43:51.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:2 cluster1 2022-04-10T01:43:51.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:2 2022-04-10T01:43:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:43:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2022-04-10T01:43:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:2 2022-04-10T01:43:51.819 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:43:51.819 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:43:51.819 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:51.819 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:43:51.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 2 = cluster1:2 ']' 2022-04-10T01:43:51.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:43:51.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:43:51.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=2 2022-04-10T01:43:51.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=2 2022-04-10T01:43:51.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:43:51.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:51.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:51.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:43:51.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2022-04-10T01:43:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:43:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:43:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:43:51.827 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_49 2022-04-10T01:43:51.929 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2022-04-10T01:43:51.929 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:43:51.930 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:43:51.933 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:43:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_49 2022-04-10T01:43:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:3 2022-04-10T01:43:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:43:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:43:51.935 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:3 cluster instance 2022-04-10T01:43:51.935 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:3 2022-04-10T01:43:51.935 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:43:51.935 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:43:51.936 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:43:51.936 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=3 2022-04-10T01:43:51.936 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 3 = cluster1:3 ']' 2022-04-10T01:43:51.936 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:43:51.936 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:51.937 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=3 2022-04-10T01:43:51.937 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=3 2022-04-10T01:43:51.937 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:43:51.937 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:43:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:43:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:3 cluster1 2022-04-10T01:43:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:3 2022-04-10T01:43:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:43:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2022-04-10T01:43:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:3 2022-04-10T01:43:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:43:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:43:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=3 2022-04-10T01:43:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 3 = cluster1:3 ']' 2022-04-10T01:43:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:43:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:43:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=3 2022-04-10T01:43:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=3 2022-04-10T01:43:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:43:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:43:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2022-04-10T01:43:51.948 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:43:51.948 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:43:51.949 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2022-04-10T01:43:51.949 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_49 2022-04-10T01:43:52.060 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:43:52.061 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:43:52.061 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:43:52.062 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:43:52.062 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:43:52.067 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:43:52.067 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:43:52.067 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:52.067 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_49 2022-04-10T01:43:52.068 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:43:52.068 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:52.068 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:52.068 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_49 2022-04-10T01:43:52.069 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:43:52.069 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:52.069 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:52.069 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:52.069 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:43:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:43:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_49 2022-04-10T01:43:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:43:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:43:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_49 2022-04-10T01:43:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:43:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:52.073 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:52.073 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:52.073 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-image_49.status 2022-04-10T01:43:52.073 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-image_49.status 2022-04-10T01:43:52.074 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:43:52.074 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_49.status 2022-04-10T01:43:52.074 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-image_49.status 2022-04-10T01:43:52.074 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-image_49.status 2022-04-10T01:43:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:43:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_49.status 2022-04-10T01:43:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror image_49 2022-04-10T01:43:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_49 2022-04-10T01:43:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:43:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_49 2022-04-10T01:43:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 263 2022-04-10T01:43:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_49 /tmp/tmp.rgwz74WspX/cluster2-mirror-image_49.status 2022-04-10T01:43:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_49 2022-04-10T01:43:52.628 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_49.status 2022-04-10T01:43:52.628 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:43:52.628 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:52.628 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:52.629 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:52.629 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_49 --format xml 2022-04-10T01:43:52.629 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:43:52.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:43:52.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=263 2022-04-10T01:43:52.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:43:52.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:43:52.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:43:52.872 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_49.status 2022-04-10T01:43:52.872 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:52.872 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:52.873 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_49 2022-04-10T01:43:52.873 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_49.status 2022-04-10T01:43:52.873 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:52.873 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:52.874 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:52.874 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:52.874 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2022-04-10T01:43:52.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:43:52.920 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:43:52.920 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:43:52.921 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:43:52.921 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:43:52.922 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:43:52.923 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:43:52.923 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 263 = '' 2022-04-10T01:43:52.923 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:43:52.923 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:43:53.324 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_49.status 2022-04-10T01:43:53.324 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:53.324 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_49 2022-04-10T01:43:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_49.status 2022-04-10T01:43:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:53.326 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:53.326 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2022-04-10T01:43:53.326 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:43:53.371 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:43:53.372 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:43:53.373 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:43:53.373 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:43:53.374 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:43:53.374 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:43:53.375 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 263 = '' 2022-04-10T01:43:53.375 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:43:53.375 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:43:54.176 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_49.status 2022-04-10T01:43:54.176 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:54.177 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:54.177 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_49 2022-04-10T01:43:54.177 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_49.status 2022-04-10T01:43:54.177 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:54.177 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:54.178 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:54.178 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:54.178 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2022-04-10T01:43:54.178 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:43:54.217 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:43:54.221 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=263 2022-04-10T01:43:54.221 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 263 = 263 2022-04-10T01:43:54.221 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:54.222 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror image_49 snap 2022-04-10T01:43:54.222 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:54.222 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:54.222 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_49 2022-04-10T01:43:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:43:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:43:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:54.224 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:54.224 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:43:54.224 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:43:55.221 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/image_49@snap 2022-04-10T01:43:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_49': 2022-04-10T01:43:55.265 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:43:55.265 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:43:55.265 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 2 2022-04-10T01:43:55.265 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 10e4c92cc126 2022-04-10T01:43:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.10e4c92cc126 2022-04-10T01:43:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:43:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:43:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:43:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:43:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:36:58 2022 2022-04-10T01:43:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:36:58 2022 2022-04-10T01:43:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:36:58 2022 2022-04-10T01:43:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:43:55.268 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:43:55.268 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:43:55.268 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: dadd4ce2-bc41-4878-8215-b7901a7d9b7c 2022-04-10T01:43:55.268 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:43:55.270 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:55.270 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror image_49 snap 2022-04-10T01:43:55.270 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:55.270 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_49 2022-04-10T01:43:55.271 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:43:55.271 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:43:55.271 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_49.export 2022-04-10T01:43:55.271 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_49.export 2022-04-10T01:43:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_49.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_49.export 2022-04-10T01:43:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export image_49@snap /tmp/tmp.rgwz74WspX/cluster2-mirror-image_49.export 2022-04-10T01:43:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:43:55.453 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export image_49@snap /tmp/tmp.rgwz74WspX/cluster1-mirror-image_49.export 2022-04-10T01:43:55.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:43:55.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-image_49.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_49.export 2022-04-10T01:43:55.750 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_49.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_49.export 2022-04-10T01:43:55.788 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:55.788 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:43:55.788 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_50 2022-04-10T01:43:55.788 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_snap cluster2 mirror image_50 snap 2022-04-10T01:43:55.789 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:55.789 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:55.789 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_50 2022-04-10T01:43:55.789 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:43:55.790 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror snap create image_50@snap --debug-rbd=20 --debug-journaler=20 2022-04-10T01:43:57.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_started cluster1 mirror image_50 2022-04-10T01:43:57.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_50 2022-04-10T01:43:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:57.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:57.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:57.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:57.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_replay_state cluster1 mirror image_50 started 2022-04-10T01:43:57.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:57.647 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:57.647 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_50 2022-04-10T01:43:57.647 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=started 2022-04-10T01:43:57.647 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:43:57.648 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:57.648 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:57.648 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:57.648 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:57.648 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:43:57.649 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:43:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_replay_state cluster1 mirror image_50 started 2022-04-10T01:43:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_50 2022-04-10T01:43:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=started 2022-04-10T01:43:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_result 2022-04-10T01:43:58.648 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=stopped 2022-04-10T01:43:58.648 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:58.648 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:58.648 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_50 2022-04-10T01:43:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep -i state 2022-04-10T01:43:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_instance=cluster1 2022-04-10T01:43:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:43:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:43:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance=cluster1 2022-04-10T01:43:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local loop_instance 2022-04-10T01:43:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:43:58.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:43:58.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:58.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:43:58.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2022-04-10T01:43:58.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sleep 0 2022-04-10T01:43:58.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 '!=' cluster1 ']' 2022-04-10T01:43:58.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ seq 0 3 2022-04-10T01:43:58.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:43:58.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_50 2022-04-10T01:43:58.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:0 2022-04-10T01:43:58.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:43:58.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:43:58.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:0 cluster instance 2022-04-10T01:43:58.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:0 2022-04-10T01:43:58.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:43:58.658 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:43:58.658 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:43:58.658 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:43:58.658 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 0 = cluster1:0 ']' 2022-04-10T01:43:58.659 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:43:58.659 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:58.659 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=0 2022-04-10T01:43:58.659 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:43:58.659 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:43:58.660 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:43:58.660 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:58.660 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:43:58.660 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:0 cluster1 2022-04-10T01:43:58.660 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:0 2022-04-10T01:43:58.661 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:58.661 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:43:58.661 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2022-04-10T01:43:58.661 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:0 2022-04-10T01:43:58.662 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:43:58.662 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:43:58.662 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:58.662 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=0 2022-04-10T01:43:58.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 0 = cluster1:0 ']' 2022-04-10T01:43:58.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:43:58.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:43:58.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=0 2022-04-10T01:43:58.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=0 2022-04-10T01:43:58.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:43:58.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:58.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:58.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:43:58.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2022-04-10T01:43:58.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:43:58.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:43:58.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2022-04-10T01:43:58.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_50 2022-04-10T01:43:58.779 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2022-04-10T01:43:58.780 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:43:58.780 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:43:58.784 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:43:58.784 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_50 2022-04-10T01:43:58.784 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:1 2022-04-10T01:43:58.785 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:43:58.785 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:43:58.785 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:1 cluster instance 2022-04-10T01:43:58.785 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:1 2022-04-10T01:43:58.785 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:43:58.786 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:43:58.786 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:43:58.786 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=1 2022-04-10T01:43:58.786 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 1 = cluster1:1 ']' 2022-04-10T01:43:58.787 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:43:58.787 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:58.787 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=1 2022-04-10T01:43:58.787 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:43:58.788 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:43:58.788 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:43:58.788 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:58.788 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:43:58.788 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:1 cluster1 2022-04-10T01:43:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:1 2022-04-10T01:43:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:43:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2022-04-10T01:43:58.790 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:1 2022-04-10T01:43:58.790 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:43:58.790 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:43:58.790 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:58.790 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=1 2022-04-10T01:43:58.791 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 1 = cluster1:1 ']' 2022-04-10T01:43:58.791 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:43:58.791 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:43:58.791 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=1 2022-04-10T01:43:58.792 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=1 2022-04-10T01:43:58.792 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:43:58.792 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:58.792 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:58.792 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:43:58.793 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2022-04-10T01:43:58.798 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:43:58.799 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:43:58.799 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2022-04-10T01:43:58.799 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_50 2022-04-10T01:43:58.900 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2022-04-10T01:43:58.900 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd mirror status : get status for rbd mirror 2022-04-10T01:43:58.900 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:admin_socket: invalid command 2022-04-10T01:43:58.904 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2022-04-10T01:43:58.905 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_50 2022-04-10T01:43:58.905 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1:2 2022-04-10T01:43:58.905 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ shift 2022-04-10T01:43:58.905 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance 2022-04-10T01:43:58.905 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set_cluster_instance cluster1:2 cluster instance 2022-04-10T01:43:58.906 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local val=cluster1:2 2022-04-10T01:43:58.906 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster_var_name=cluster 2022-04-10T01:43:58.906 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local instance_var_name=instance 2022-04-10T01:43:58.906 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:43:58.906 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=2 2022-04-10T01:43:58.907 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 2 = cluster1:2 ']' 2022-04-10T01:43:58.907 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval cluster=cluster1 2022-04-10T01:43:58.907 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:58.907 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ eval instance=2 2022-04-10T01:43:58.908 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:43:58.908 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:43:58.908 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:43:58.908 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:58.909 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:43:58.909 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ daemon_asok_file cluster1:2 cluster1 2022-04-10T01:43:58.909 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local local_cluster=cluster1:2 2022-04-10T01:43:58.909 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:58.909 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance 2022-04-10T01:43:58.910 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2022-04-10T01:43:58.910 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local val=cluster1:2 2022-04-10T01:43:58.910 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster_var_name=local_cluster 2022-04-10T01:43:58.910 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local instance_var_name=instance 2022-04-10T01:43:58.911 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ cluster=cluster1 2022-04-10T01:43:58.911 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ instance=2 2022-04-10T01:43:58.911 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' 2 = cluster1:2 ']' 2022-04-10T01:43:58.911 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval local_cluster=cluster1 2022-04-10T01:43:58.911 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ local_cluster=cluster1 2022-04-10T01:43:58.912 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ eval instance=2 2022-04-10T01:43:58.912 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ instance=2 2022-04-10T01:43:58.912 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ can_execute cluster1 2022-04-10T01:43:58.912 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ local cluster=cluster1 2022-04-10T01:43:58.912 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ return 0 2022-04-10T01:43:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2022-04-10T01:43:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:43:58.919 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:43:58.919 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2022-04-10T01:43:58.919 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_50 2022-04-10T01:43:59.030 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:43:59.031 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status_result=' "state": "Replaying"' 2022-04-10T01:43:59.031 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ' "state": "Replaying"' 2022-04-10T01:43:59.031 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ grep -i Replaying 2022-04-10T01:43:59.032 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: "state": "Replaying" 2022-04-10T01:43:59.036 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=started 2022-04-10T01:43:59.036 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test started = started 2022-04-10T01:43:59.036 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:59.036 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_50 2022-04-10T01:43:59.037 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:43:59.037 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:59.037 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:59.037 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_50 2022-04-10T01:43:59.038 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:43:59.038 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:59.038 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:59.038 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:59.039 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = journal ']' 2022-04-10T01:43:59.039 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' snapshot = snapshot ']' 2022-04-10T01:43:59.039 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_50 2022-04-10T01:43:59.039 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_cluster=cluster1 2022-04-10T01:43:59.040 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local remote_cluster=cluster2 2022-04-10T01:43:59.040 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:59.040 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_50 2022-04-10T01:43:59.040 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:43:59.041 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:43:59.041 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:59.041 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:59.041 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:43:59.041 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster2-mirror-image_50.status 2022-04-10T01:43:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster2-mirror-image_50.status 2022-04-10T01:43:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:43:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local status_log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_50.status 2022-04-10T01:43:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ mkfname cluster1-mirror-image_50.status 2022-04-10T01:43:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo cluster1-mirror-image_50.status 2022-04-10T01:43:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -e 's|[/ ]|_|g' 2022-04-10T01:43:59.044 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local local_status_log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_50.status 2022-04-10T01:43:59.044 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ mirror_image_snapshot cluster2 mirror image_50 2022-04-10T01:43:59.044 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:59.044 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:43:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_50 2022-04-10T01:43:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:43:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:43:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:43:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:43:59.046 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_50 2022-04-10T01:44:00.631 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Snapshot ID: 267 2022-04-10T01:44:00.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_50 /tmp/tmp.rgwz74WspX/cluster2-mirror-image_50.status 2022-04-10T01:44:00.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:00.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:00.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_50 2022-04-10T01:44:00.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_50.status 2022-04-10T01:44:00.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:00.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:00.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:00.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:00.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_50 --format xml 2022-04-10T01:44:00.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:44:00.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/id 2022-04-10T01:44:00.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snapshot_id=267 2022-04-10T01:44:00.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:44:00.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:44:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.2 2022-04-10T01:44:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_50.status 2022-04-10T01:44:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:44:00.892 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:00.892 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_50 2022-04-10T01:44:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_50.status 2022-04-10T01:44:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:44:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:44:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:00.895 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2022-04-10T01:44:00.895 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:44:00.937 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:44:00.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:44:00.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:44:00.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:44:00.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:44:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:44:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 267 = '' 2022-04-10T01:44:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:44:00.941 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.4 2022-04-10T01:44:01.342 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_50.status 2022-04-10T01:44:01.342 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:44:01.342 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:01.342 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_50 2022-04-10T01:44:01.343 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_50.status 2022-04-10T01:44:01.343 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:44:01.343 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:44:01.343 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:01.343 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:01.344 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2022-04-10T01:44:01.344 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:44:01.389 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:44:01.390 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:44:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:-:1.1: Document is empty 2022-04-10T01:44:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: 2022-04-10T01:44:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:^ 2022-04-10T01:44:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id= 2022-04-10T01:44:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 267 = '' 2022-04-10T01:44:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2022-04-10T01:44:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.8 2022-04-10T01:44:02.194 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.rgwz74WspX/cluster1-mirror-image_50.status 2022-04-10T01:44:02.194 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:44:02.194 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:02.194 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_50 2022-04-10T01:44:02.195 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local log=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_50.status 2022-04-10T01:44:02.195 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:44:02.195 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:44:02.195 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:02.196 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:02.196 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2022-04-10T01:44:02.196 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2022-04-10T01:44:02.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2022-04-10T01:44:02.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local primary_snapshot_id=267 2022-04-10T01:44:02.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 267 = 267 2022-04-10T01:44:02.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:02.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_snap_present cluster1 mirror image_50 snap 2022-04-10T01:44:02.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:44:02.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:02.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_50 2022-04-10T01:44:02.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:44:02.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:44:02.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:44:02.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:44:02.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:02.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:02.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2022-04-10T01:44:02.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 1 2022-04-10T01:44:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 info mirror/image_50@snap 2022-04-10T01:44:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_50': 2022-04-10T01:44:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 128 MiB in 32 objects 2022-04-10T01:44:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:44:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 2 2022-04-10T01:44:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 11338f3598a4 2022-04-10T01:44:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.11338f3598a4 2022-04-10T01:44:03.283 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:44:03.283 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock, non-primary 2022-04-10T01:44:03.283 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:44:03.283 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:44:03.284 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:37:12 2022 2022-04-10T01:44:03.284 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:37:12 2022 2022-04-10T01:44:03.284 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:37:12 2022 2022-04-10T01:44:03.284 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: protected: False 2022-04-10T01:44:03.284 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:44:03.285 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:44:03.285 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: a17938e9-2a1e-432d-9eca-f1aacf780959 2022-04-10T01:44:03.285 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: false 2022-04-10T01:44:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ compare_image_snaps mirror image_50 snap 2022-04-10T01:44:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_50 2022-04-10T01:44:03.288 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap_name=snap 2022-04-10T01:44:03.288 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local ret=0 2022-04-10T01:44:03.288 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local rmt_export=/tmp/tmp.rgwz74WspX/cluster2-mirror-image_50.export 2022-04-10T01:44:03.288 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local loc_export=/tmp/tmp.rgwz74WspX/cluster1-mirror-image_50.export 2022-04-10T01:44:03.288 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_50.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_50.export 2022-04-10T01:44:03.289 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 -p mirror export image_50@snap /tmp/tmp.rgwz74WspX/cluster2-mirror-image_50.export 2022-04-10T01:44:03.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:44:03.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror export image_50@snap /tmp/tmp.rgwz74WspX/cluster1-mirror-image_50.export 2022-04-10T01:44:03.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2022-04-10T01:44:03.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cmp /tmp/tmp.rgwz74WspX/cluster2-mirror-image_50.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_50.export 2022-04-10T01:44:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rm -f /tmp/tmp.rgwz74WspX/cluster2-mirror-image_50.export /tmp/tmp.rgwz74WspX/cluster1-mirror-image_50.export 2022-04-10T01:44:04.464 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:04.465 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ testlog 'TEST: delete many images' 2022-04-10T01:44:04.465 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ tee -a /tmp/tmp.rgwz74WspX/rbd-mirror.test.log 2022-04-10T01:44:04.465 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ date '+%F %T' 2022-04-10T01:44:04.466 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo 2022-04-10 01:44:04 TEST: delete many images 2022-04-10T01:44:04.466 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:2022-04-10 01:44:04 TEST: delete many images 2022-04-10T01:44:04.466 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ seq 1 50 2022-04-10T01:44:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:44:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_1 2022-04-10T01:44:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror image_1 snap 2022-04-10T01:44:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_1 2022-04-10T01:44:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap 2022-04-10T01:44:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:04.471 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/image_1@snap 2022-04-10T01:44:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:44:04.544 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_1 2022-04-10T01:44:04.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:04.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:04.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_1 2022-04-10T01:44:04.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:04.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:04.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:04.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:04.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:44:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:44:04.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_1 2022-04-10T01:44:04.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:04.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:04.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_1 2022-04-10T01:44:04.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:04.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:04.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:04.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:04.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2022-04-10T01:44:04.587 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2022-04-10T01:44:04.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi043.stderr:2022-04-10T01:44:04.696+0000 7fc2c294c700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55b0c01111e0 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:44:04.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi043.stderr:2022-04-10T01:44:04.696+0000 7fc2c9159700 -1 rbd::mirror::ImageReplayer: 0x55b0c00e5400 [3/acfa4173-7a18-4ae9-975a-94af9fdf80e4] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T01:44:05.763 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-10T01:44:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:44:05.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_2 2022-04-10T01:44:05.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror image_2 snap 2022-04-10T01:44:05.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:05.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:05.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_2 2022-04-10T01:44:05.769 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap 2022-04-10T01:44:05.769 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:05.769 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:05.769 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:05.770 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:05.770 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/image_2@snap 2022-04-10T01:44:06.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:44:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_2 2022-04-10T01:44:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_2 2022-04-10T01:44:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:44:06.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:44:06.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_2 2022-04-10T01:44:06.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_2 2022-04-10T01:44:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:06.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:06.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2022-04-10T01:44:06.700 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2022-04-10T01:44:06.808 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr:2022-04-10T01:44:06.806+0000 7f9ecc7f5700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f1a10a0680 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:44:06.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr:2022-04-10T01:44:06.806+0000 7f9ed3002700 -1 rbd::mirror::ImageReplayer: 0x55f1a0b2b000 [3/a0c16b90-9fb2-4d3c-834f-4b590db7c01d] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T01:44:07.772 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-10T01:44:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:44:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_3 2022-04-10T01:44:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror image_3 snap 2022-04-10T01:44:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_3 2022-04-10T01:44:07.778 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap 2022-04-10T01:44:07.778 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:07.778 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:07.778 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:07.779 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:07.779 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/image_3@snap 2022-04-10T01:44:08.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:44:08.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_3 2022-04-10T01:44:08.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:08.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:08.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_3 2022-04-10T01:44:08.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:08.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:08.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:08.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:08.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:44:08.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:44:08.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_3 2022-04-10T01:44:08.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:08.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:08.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_3 2022-04-10T01:44:08.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:08.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:08.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:08.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:08.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2022-04-10T01:44:08.731 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2022-04-10T01:44:08.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr:2022-04-10T01:44:08.875+0000 7f9ecbff4700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f1a10651e0 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:44:08.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr:2022-04-10T01:44:08.875+0000 7f9ed2801700 -1 rbd::mirror::ImageReplayer: 0x55f1a0b2b200 [3/6d595122-4619-4d21-a653-5b0cde552987] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T01:44:09.637 INFO:tasks.ceph.cluster2.osd.2.smithi043.stderr:2022-04-10T01:44:09.634+0000 7f8be0453700 -1 reset not still connected to 0x55f2b62e2c30 2022-04-10T01:44:09.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-10T01:44:09.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:09.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:44:09.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_4 2022-04-10T01:44:09.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror image_4 snap 2022-04-10T01:44:09.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:09.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:09.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_4 2022-04-10T01:44:09.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap 2022-04-10T01:44:09.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:09.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:09.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:09.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:09.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/image_4@snap 2022-04-10T01:44:10.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:44:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_4 2022-04-10T01:44:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:10.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:10.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_4 2022-04-10T01:44:10.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:10.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:10.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:10.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:10.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:44:10.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:44:10.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_4 2022-04-10T01:44:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_4 2022-04-10T01:44:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2022-04-10T01:44:10.713 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2022-04-10T01:44:10.855 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi043.stderr:2022-04-10T01:44:10.853+0000 7fc2c294c700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55b0c05401a0 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:44:10.856 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi043.stderr:2022-04-10T01:44:10.853+0000 7fc2c8958700 -1 rbd::mirror::ImageReplayer: 0x55b0be5d2a00 [3/d4a4b534-85cd-4706-8f4e-137718b81a97] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T01:44:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-10T01:44:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:44:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_5 2022-04-10T01:44:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror image_5 snap 2022-04-10T01:44:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_5 2022-04-10T01:44:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap 2022-04-10T01:44:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/image_5@snap 2022-04-10T01:44:12.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:44:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_5 2022-04-10T01:44:12.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:12.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:12.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_5 2022-04-10T01:44:12.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:12.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:12.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:12.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:12.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:44:12.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:44:12.677 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_5 2022-04-10T01:44:12.677 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:12.677 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:12.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_5 2022-04-10T01:44:12.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:12.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:12.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:12.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:12.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2022-04-10T01:44:12.723 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2022-04-10T01:44:12.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi043.stderr:2022-04-10T01:44:12.860+0000 7f4f5dabb700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5603b90dd6c0 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:44:12.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi043.stderr:2022-04-10T01:44:12.860+0000 7f4f642c8700 -1 rbd::mirror::ImageReplayer: 0x5603b8df3e00 [3/04a20513-9ad9-463b-ad2e-0f637bb8e1de] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T01:44:13.795 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-10T01:44:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:44:13.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_6 2022-04-10T01:44:13.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror image_6 snap 2022-04-10T01:44:13.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:13.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:13.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_6 2022-04-10T01:44:13.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap 2022-04-10T01:44:13.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:13.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:13.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:13.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:13.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/image_6@snap 2022-04-10T01:44:14.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:44:14.567 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_6 2022-04-10T01:44:14.567 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:14.567 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:14.568 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_6 2022-04-10T01:44:14.568 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:14.568 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:14.568 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:14.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:14.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:44:14.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:44:14.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_6 2022-04-10T01:44:14.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:14.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:14.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_6 2022-04-10T01:44:14.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2022-04-10T01:44:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2022-04-10T01:44:14.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi043.stderr:2022-04-10T01:44:14.717+0000 7fe8bb98b700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55b34a3aa9c0 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:44:14.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi043.stderr:2022-04-10T01:44:14.717+0000 7fe8c2198700 -1 rbd::mirror::ImageReplayer: 0x55b349fe4600 [3/a2f508f1-f607-46da-a720-0b026055c5d9] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T01:44:15.795 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-10T01:44:15.799 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:15.799 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:44:15.799 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_7 2022-04-10T01:44:15.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror image_7 snap 2022-04-10T01:44:15.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:15.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:15.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_7 2022-04-10T01:44:15.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap 2022-04-10T01:44:15.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:15.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:15.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/image_7@snap 2022-04-10T01:44:16.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:44:16.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_7 2022-04-10T01:44:16.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_7 2022-04-10T01:44:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:44:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:44:16.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_7 2022-04-10T01:44:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_7 2022-04-10T01:44:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:16.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2022-04-10T01:44:16.731 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2022-04-10T01:44:16.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi043.stderr:2022-04-10T01:44:16.871+0000 7fc2c214b700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55b0c0540340 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:44:16.874 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi043.stderr:2022-04-10T01:44:16.871+0000 7fc2c8958700 -1 rbd::mirror::ImageReplayer: 0x55b0c0273800 [3/551577c0-7aad-4cdb-945f-ca54529923ea] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T01:44:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-10T01:44:17.811 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:44:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_8 2022-04-10T01:44:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror image_8 snap 2022-04-10T01:44:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:17.813 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:17.813 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_8 2022-04-10T01:44:17.813 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap 2022-04-10T01:44:17.813 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:17.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:17.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:17.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:17.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/image_8@snap 2022-04-10T01:44:18.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:44:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_8 2022-04-10T01:44:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:18.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:18.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_8 2022-04-10T01:44:18.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:18.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:18.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:18.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:18.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:44:18.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:44:18.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_8 2022-04-10T01:44:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_8 2022-04-10T01:44:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:18.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:18.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2022-04-10T01:44:18.740 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2022-04-10T01:44:18.855 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr:2022-04-10T01:44:18.852+0000 7f9ecbff4700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f1a10a0820 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:44:18.855 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr:2022-04-10T01:44:18.852+0000 7f9ed3002700 -1 rbd::mirror::ImageReplayer: 0x55f1a0b2b400 [3/77c1be68-fcf7-4bbb-bf06-177e6f3d4112] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T01:44:19.199 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.34973418960922) > weight (0.25) 2022-04-10T01:44:19.200 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7235296608254395) > weight (0.25) 2022-04-10T01:44:19.200 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.35744599386376485) > weight (0.25) 2022-04-10T01:44:19.200 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.5993621995340167) > weight (0.25) 2022-04-10T01:44:19.200 INFO:tasks.rbd_mirror_thrash:waiting for 94 secs before thrashing 2022-04-10T01:44:19.882 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-10T01:44:19.886 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:19.886 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:44:19.887 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_9 2022-04-10T01:44:19.887 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror image_9 snap 2022-04-10T01:44:19.887 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:19.887 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:19.888 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_9 2022-04-10T01:44:19.888 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap 2022-04-10T01:44:19.888 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:19.888 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:19.888 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/image_9@snap 2022-04-10T01:44:20.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:44:20.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_9 2022-04-10T01:44:20.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:20.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:20.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_9 2022-04-10T01:44:20.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:20.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:20.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:20.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:20.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:44:20.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:44:20.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_9 2022-04-10T01:44:20.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:20.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:20.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_9 2022-04-10T01:44:20.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:20.698 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:20.698 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:20.698 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:20.699 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2022-04-10T01:44:20.744 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2022-04-10T01:44:20.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi043.stderr:2022-04-10T01:44:20.883+0000 7f4f5d2ba700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5603b918eb60 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:44:20.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi043.stderr:2022-04-10T01:44:20.883+0000 7f4f63ac7700 -1 rbd::mirror::ImageReplayer: 0x5603b698a600 [3/0661b513-be8b-4e37-9a6b-b32c1a1daf4e] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T01:44:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-10T01:44:21.819 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:21.819 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:44:21.819 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_10 2022-04-10T01:44:21.819 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror image_10 snap 2022-04-10T01:44:21.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:21.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:21.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_10 2022-04-10T01:44:21.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap 2022-04-10T01:44:21.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:21.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:21.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:21.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:21.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/image_10@snap 2022-04-10T01:44:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:44:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_10 2022-04-10T01:44:22.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:22.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:22.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_10 2022-04-10T01:44:22.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:22.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:22.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:22.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:22.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:44:22.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:44:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_10 2022-04-10T01:44:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_10 2022-04-10T01:44:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2022-04-10T01:44:22.756 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2022-04-10T01:44:22.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr:2022-04-10T01:44:22.886+0000 7f9ecbff4700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f1a10b16c0 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:44:22.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr:2022-04-10T01:44:22.886+0000 7f9ed3002700 -1 rbd::mirror::ImageReplayer: 0x55f1a0b2b600 [3/e9d2f488-240a-4db7-8e1f-b458c15d12a8] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T01:44:23.829 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-10T01:44:23.833 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:23.833 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:44:23.833 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_11 2022-04-10T01:44:23.834 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror image_11 snap 2022-04-10T01:44:23.834 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:23.834 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:23.834 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_11 2022-04-10T01:44:23.834 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap 2022-04-10T01:44:23.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:23.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:23.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:23.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:23.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/image_11@snap 2022-04-10T01:44:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:44:24.568 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_11 2022-04-10T01:44:24.568 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:24.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:24.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_11 2022-04-10T01:44:24.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:24.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:24.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:24.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:24.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:44:24.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:44:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_11 2022-04-10T01:44:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_11 2022-04-10T01:44:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_11 2022-04-10T01:44:24.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_11 2022-04-10T01:44:24.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr:2022-04-10T01:44:24.720+0000 7f9ecbff4700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f1a10b1860 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:44:24.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr:2022-04-10T01:44:24.720+0000 7f9ed2801700 -1 rbd::mirror::ImageReplayer: 0x55f1a0b28200 [3/ff2a6efd-2c82-4070-8950-c9c7f1385660] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T01:44:25.827 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-10T01:44:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:25.832 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:44:25.832 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_12 2022-04-10T01:44:25.832 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror image_12 snap 2022-04-10T01:44:25.832 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:25.833 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:25.833 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_12 2022-04-10T01:44:25.833 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap 2022-04-10T01:44:25.833 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:25.833 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:25.834 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:25.834 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:25.834 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/image_12@snap 2022-04-10T01:44:26.703 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:44:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_12 2022-04-10T01:44:26.719 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:26.719 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:26.719 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_12 2022-04-10T01:44:26.719 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:26.720 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:26.720 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:26.720 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:26.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:44:26.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:44:26.722 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_12 2022-04-10T01:44:26.722 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:26.723 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:26.723 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_12 2022-04-10T01:44:26.723 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:26.723 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:26.724 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:26.724 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:26.724 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_12 2022-04-10T01:44:26.765 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_12 2022-04-10T01:44:26.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi043.stderr:2022-04-10T01:44:26.905+0000 7fe8bb98b700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55b34a3aab60 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:44:26.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi043.stderr:2022-04-10T01:44:26.906+0000 7fe8c2198700 -1 rbd::mirror::ImageReplayer: 0x55b349e43600 [3/672aa5fe-c483-4bea-94d8-2b504c745764] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T01:44:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-10T01:44:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:27.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:44:27.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_13 2022-04-10T01:44:27.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror image_13 snap 2022-04-10T01:44:27.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:27.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:27.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_13 2022-04-10T01:44:27.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap 2022-04-10T01:44:27.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:27.837 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:27.837 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:27.837 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:27.837 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/image_13@snap 2022-04-10T01:44:28.706 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:44:28.717 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_13 2022-04-10T01:44:28.718 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:28.718 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:28.718 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_13 2022-04-10T01:44:28.718 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:44:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:44:28.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_13 2022-04-10T01:44:28.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:28.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:28.722 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_13 2022-04-10T01:44:28.722 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:28.722 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:28.723 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:28.723 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:28.723 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_13 2022-04-10T01:44:28.774 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_13 2022-04-10T01:44:28.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi043.stderr:2022-04-10T01:44:28.874+0000 7fc2c214b700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55b0c05404e0 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:44:28.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi043.stderr:2022-04-10T01:44:28.874+0000 7fc2c8958700 -1 rbd::mirror::ImageReplayer: 0x55b0c00e5200 [3/b7fe287d-2790-4653-8a15-70f6e8da8e9f] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T01:44:29.636 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr:2022-04-10T01:44:29.631+0000 7f52fc8d6700 -1 reset not still connected to 0x5600fe772ea0 2022-04-10T01:44:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-10T01:44:29.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:29.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:44:29.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_14 2022-04-10T01:44:29.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror image_14 snap 2022-04-10T01:44:29.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:29.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:29.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_14 2022-04-10T01:44:29.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap 2022-04-10T01:44:29.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:29.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:29.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:29.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:29.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/image_14@snap 2022-04-10T01:44:30.713 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:44:30.722 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_14 2022-04-10T01:44:30.722 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:30.723 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:30.723 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_14 2022-04-10T01:44:30.723 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:30.723 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:30.724 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:30.724 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:30.724 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:44:30.724 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:44:30.725 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_14 2022-04-10T01:44:30.726 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:30.726 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:30.726 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_14 2022-04-10T01:44:30.726 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:30.727 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:30.727 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:30.727 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:30.727 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_14 2022-04-10T01:44:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_14 2022-04-10T01:44:30.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi043.stderr:2022-04-10T01:44:30.907+0000 7fe8bb98b700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55b3479781a0 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:44:30.911 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi043.stderr:2022-04-10T01:44:30.908+0000 7fe8c2999700 -1 rbd::mirror::ImageReplayer: 0x55b349e43e00 [3/9daee277-0f48-4a3a-be63-d5fc81096bc5] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T01:44:31.852 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-10T01:44:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:44:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_15 2022-04-10T01:44:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror image_15 snap 2022-04-10T01:44:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:31.858 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_15 2022-04-10T01:44:31.858 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap 2022-04-10T01:44:31.858 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:31.858 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:31.859 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:31.859 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:31.859 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/image_15@snap 2022-04-10T01:44:32.726 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:44:32.736 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_15 2022-04-10T01:44:32.736 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:32.736 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_15 2022-04-10T01:44:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:44:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:44:32.739 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_15 2022-04-10T01:44:32.739 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:32.740 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:32.740 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_15 2022-04-10T01:44:32.741 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:32.741 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:32.742 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:32.742 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:32.743 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_15 2022-04-10T01:44:32.782 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_15 2022-04-10T01:44:32.893 INFO:tasks.ceph.cluster1.osd.2.smithi005.stderr:2022-04-10T01:44:32.889+0000 7f0c60f96700 -1 reset not still connected to 0x564c18c4b110 2022-04-10T01:44:32.893 INFO:tasks.ceph.cluster1.osd.2.smithi005.stderr:2022-04-10T01:44:32.889+0000 7f0c60f96700 -1 reset not still connected to 0x564c19869790 2022-04-10T01:44:32.922 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi043.stderr:2022-04-10T01:44:32.919+0000 7f4f5d2ba700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5603b90dda00 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:44:32.922 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi043.stderr:2022-04-10T01:44:32.919+0000 7f4f642c8700 -1 rbd::mirror::ImageReplayer: 0x5603b8df3200 [3/1de7d6c4-ab1c-49a3-8053-b58d65c37430] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T01:44:33.853 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-10T01:44:33.856 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:33.856 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:44:33.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_16 2022-04-10T01:44:33.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror image_16 snap 2022-04-10T01:44:33.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:33.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_16 2022-04-10T01:44:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap 2022-04-10T01:44:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:33.859 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:33.859 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:33.859 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/image_16@snap 2022-04-10T01:44:34.551 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:44:34.562 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_16 2022-04-10T01:44:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_16 2022-04-10T01:44:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:34.566 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:44:34.566 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:44:34.567 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_16 2022-04-10T01:44:34.567 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:34.568 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:34.568 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_16 2022-04-10T01:44:34.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:34.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:34.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:34.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:34.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_16 2022-04-10T01:44:34.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_16 2022-04-10T01:44:34.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi043.stderr:2022-04-10T01:44:34.720+0000 7fe8bc18c700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55b347978000 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:44:34.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi043.stderr:2022-04-10T01:44:34.720+0000 7fe8c2999700 -1 rbd::mirror::ImageReplayer: 0x55b349e43400 [3/47fb89d8-4dfe-49df-833b-c0b0fd34dfb9] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T01:44:35.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-10T01:44:35.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:35.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:44:35.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_17 2022-04-10T01:44:35.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror image_17 snap 2022-04-10T01:44:35.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:35.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:35.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_17 2022-04-10T01:44:35.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap 2022-04-10T01:44:35.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:35.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:35.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:35.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:35.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/image_17@snap 2022-04-10T01:44:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:44:36.734 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_17 2022-04-10T01:44:36.734 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_17 2022-04-10T01:44:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:44:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:44:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_17 2022-04-10T01:44:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_17 2022-04-10T01:44:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:36.740 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_17 2022-04-10T01:44:36.781 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_17 2022-04-10T01:44:36.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr:2022-04-10T01:44:36.935+0000 7f9ecc7f5700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f1a10a09c0 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:44:36.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr:2022-04-10T01:44:36.935+0000 7f9ed3002700 -1 rbd::mirror::ImageReplayer: 0x55f1a0b2b800 [3/59134803-ff09-429b-9467-2cdbc3d6d23c] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T01:44:37.854 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-10T01:44:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:44:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_18 2022-04-10T01:44:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror image_18 snap 2022-04-10T01:44:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_18 2022-04-10T01:44:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap 2022-04-10T01:44:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:37.861 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:37.861 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:37.861 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/image_18@snap 2022-04-10T01:44:38.736 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:44:38.748 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_18 2022-04-10T01:44:38.748 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:38.748 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:38.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_18 2022-04-10T01:44:38.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:38.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:38.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:38.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:38.750 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:44:38.750 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:44:38.751 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_18 2022-04-10T01:44:38.751 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:38.752 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:38.752 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_18 2022-04-10T01:44:38.752 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:38.752 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:38.753 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:38.753 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:38.753 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_18 2022-04-10T01:44:38.797 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_18 2022-04-10T01:44:38.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi043.stderr:2022-04-10T01:44:38.899+0000 7fe8bc18c700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55b347979860 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:44:38.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi043.stderr:2022-04-10T01:44:38.899+0000 7fe8c2999700 -1 rbd::mirror::ImageReplayer: 0x55b349e43200 [3/0481cc7e-2bc7-4c1c-903d-89babd0e91d4] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T01:44:39.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-10T01:44:39.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:39.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:44:39.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_19 2022-04-10T01:44:39.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror image_19 snap 2022-04-10T01:44:39.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:39.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:39.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_19 2022-04-10T01:44:39.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap 2022-04-10T01:44:39.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:39.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:39.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:39.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:39.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/image_19@snap 2022-04-10T01:44:40.731 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:44:40.748 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_19 2022-04-10T01:44:40.748 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:40.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:40.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_19 2022-04-10T01:44:40.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:40.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:40.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:40.750 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:40.750 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:44:40.750 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:44:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_19 2022-04-10T01:44:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_19 2022-04-10T01:44:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_19 2022-04-10T01:44:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_19 2022-04-10T01:44:40.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi043.stderr:2022-04-10T01:44:40.941+0000 7f4f5d2ba700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5603b90dd380 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:44:40.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi043.stderr:2022-04-10T01:44:40.941+0000 7f4f642c8700 -1 rbd::mirror::ImageReplayer: 0x5603b8df3600 [3/45a317c4-845d-40d4-b7de-aa8c3ca7ce90] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T01:44:41.872 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-10T01:44:41.876 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:41.876 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:44:41.877 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_20 2022-04-10T01:44:41.877 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror image_20 snap 2022-04-10T01:44:41.877 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:41.877 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:41.877 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_20 2022-04-10T01:44:41.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap 2022-04-10T01:44:41.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:41.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:41.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:41.879 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:41.879 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/image_20@snap 2022-04-10T01:44:42.743 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:44:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_20 2022-04-10T01:44:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_20 2022-04-10T01:44:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:42.754 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:44:42.754 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:44:42.755 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_20 2022-04-10T01:44:42.755 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:42.756 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:42.756 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_20 2022-04-10T01:44:42.756 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:42.756 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:42.757 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:42.757 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:42.757 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_20 2022-04-10T01:44:42.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_20 2022-04-10T01:44:42.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi043.stderr:2022-04-10T01:44:42.942+0000 7fe8bb98b700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55b347979d40 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:44:42.945 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi043.stderr:2022-04-10T01:44:42.942+0000 7fe8c2999700 -1 rbd::mirror::ImageReplayer: 0x55b349e43800 [3/828304b8-d69a-488c-b09a-e6569fe239bd] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T01:44:43.881 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-10T01:44:43.885 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:43.885 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:44:43.885 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_21 2022-04-10T01:44:43.885 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror image_21 snap 2022-04-10T01:44:43.886 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:43.886 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:43.886 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_21 2022-04-10T01:44:43.886 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap 2022-04-10T01:44:43.887 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:43.887 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:43.887 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:43.887 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/image_21@snap 2022-04-10T01:44:44.558 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:44:44.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_21 2022-04-10T01:44:44.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:44.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:44.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_21 2022-04-10T01:44:44.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:44.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:44.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:44.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:44.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:44:44.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:44:44.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_21 2022-04-10T01:44:44.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:44.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:44.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_21 2022-04-10T01:44:44.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_21 2022-04-10T01:44:44.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_21 2022-04-10T01:44:44.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi043.stderr:2022-04-10T01:44:44.714+0000 7fc2c214b700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55b0c0555040 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:44:44.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi043.stderr:2022-04-10T01:44:44.714+0000 7fc2c9159700 -1 rbd::mirror::ImageReplayer: 0x55b0c00e5600 [3/65c94954-3314-4812-9697-85faa0cd8459] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T01:44:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-10T01:44:45.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:45.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:44:45.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_22 2022-04-10T01:44:45.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror image_22 snap 2022-04-10T01:44:45.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:45.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:45.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_22 2022-04-10T01:44:45.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap 2022-04-10T01:44:45.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:45.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:45.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:45.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:45.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/image_22@snap 2022-04-10T01:44:46.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:44:46.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_22 2022-04-10T01:44:46.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:46.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:46.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_22 2022-04-10T01:44:46.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:46.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:46.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:46.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:46.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:44:46.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:44:46.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_22 2022-04-10T01:44:46.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:46.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:46.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_22 2022-04-10T01:44:46.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:46.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:46.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:46.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:46.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_22 2022-04-10T01:44:46.619 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_22 2022-04-10T01:44:46.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr:2022-04-10T01:44:46.724+0000 7f9ecbff4700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f1a10a0b60 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:44:46.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr:2022-04-10T01:44:46.724+0000 7f9ed3002700 -1 rbd::mirror::ImageReplayer: 0x55f19e64dc00 [3/be4c9569-5fe3-47a0-a0a1-84d380d56af1] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T01:44:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-10T01:44:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:44:47.898 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_23 2022-04-10T01:44:47.898 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror image_23 snap 2022-04-10T01:44:47.898 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:47.898 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:47.899 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_23 2022-04-10T01:44:47.899 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap 2022-04-10T01:44:47.899 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:47.899 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:47.899 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:47.900 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:47.900 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/image_23@snap 2022-04-10T01:44:48.563 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:44:48.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_23 2022-04-10T01:44:48.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:48.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:48.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_23 2022-04-10T01:44:48.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:44:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:44:48.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_23 2022-04-10T01:44:48.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:48.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:48.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_23 2022-04-10T01:44:48.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:48.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:48.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:48.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:48.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_23 2022-04-10T01:44:48.619 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_23 2022-04-10T01:44:48.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi043.stderr:2022-04-10T01:44:48.723+0000 7f4f5dabb700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5603b9090ea0 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:44:48.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi043.stderr:2022-04-10T01:44:48.723+0000 7f4f642c8700 -1 rbd::mirror::ImageReplayer: 0x5603b8df3400 [3/656c53f5-50a2-44ea-8dbc-608ddbbbe7f5] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T01:44:49.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-10T01:44:49.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:44:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_24 2022-04-10T01:44:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror image_24 snap 2022-04-10T01:44:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_24 2022-04-10T01:44:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap 2022-04-10T01:44:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/image_24@snap 2022-04-10T01:44:50.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:44:50.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_24 2022-04-10T01:44:50.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_24 2022-04-10T01:44:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:50.584 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:50.584 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:50.584 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:50.585 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:44:50.585 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:44:50.586 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_24 2022-04-10T01:44:50.586 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:50.586 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:50.587 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_24 2022-04-10T01:44:50.587 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:50.589 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:50.589 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_24 2022-04-10T01:44:50.626 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_24 2022-04-10T01:44:50.742 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi043.stderr:2022-04-10T01:44:50.739+0000 7fc2c294c700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55b0c05551e0 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:44:50.742 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi043.stderr:2022-04-10T01:44:50.739+0000 7fc2c9159700 -1 rbd::mirror::ImageReplayer: 0x55b0bfe57200 [3/a52ba6c5-cf55-4fb1-99eb-fa7caff7c3bd] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T01:44:51.705 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-10T01:44:51.709 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:51.709 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:44:51.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_25 2022-04-10T01:44:51.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror image_25 snap 2022-04-10T01:44:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_25 2022-04-10T01:44:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap 2022-04-10T01:44:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/image_25@snap 2022-04-10T01:44:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:44:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_25 2022-04-10T01:44:52.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:52.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:52.594 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_25 2022-04-10T01:44:52.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:52.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:52.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:52.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:52.595 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:44:52.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:44:52.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_25 2022-04-10T01:44:52.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:52.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:52.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_25 2022-04-10T01:44:52.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:52.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:52.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:52.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:52.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_25 2022-04-10T01:44:52.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_25 2022-04-10T01:44:52.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr:2022-04-10T01:44:52.748+0000 7f9ecbff4700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f1a10b1ba0 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:44:52.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr:2022-04-10T01:44:52.748+0000 7f9ed2801700 -1 rbd::mirror::ImageReplayer: 0x55f1a0b2ac00 [3/a8a88faf-cd63-4557-9afd-ca8d6505edb6] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T01:44:53.713 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-10T01:44:53.716 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:53.716 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:44:53.717 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_26 2022-04-10T01:44:53.717 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror image_26 snap 2022-04-10T01:44:53.717 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:53.717 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:53.718 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_26 2022-04-10T01:44:53.718 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap 2022-04-10T01:44:53.718 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:53.718 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:53.719 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:53.719 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:53.719 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/image_26@snap 2022-04-10T01:44:54.562 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:44:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_26 2022-04-10T01:44:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_26 2022-04-10T01:44:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:44:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:44:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_26 2022-04-10T01:44:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_26 2022-04-10T01:44:54.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:54.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:54.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:54.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:54.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_26 2022-04-10T01:44:54.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_26 2022-04-10T01:44:54.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr:2022-04-10T01:44:54.735+0000 7f9ecbff4700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f1a10a16c0 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:44:54.738 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr:2022-04-10T01:44:54.735+0000 7f9ed3002700 -1 rbd::mirror::ImageReplayer: 0x55f1a0b29600 [3/ef9f36c8-5f7f-4c73-8f03-3933b20566c7] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T01:44:55.712 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-10T01:44:55.716 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:55.716 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:44:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_27 2022-04-10T01:44:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror image_27 snap 2022-04-10T01:44:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_27 2022-04-10T01:44:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap 2022-04-10T01:44:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/image_27@snap 2022-04-10T01:44:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:44:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_27 2022-04-10T01:44:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_27 2022-04-10T01:44:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:56.605 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:56.605 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:44:56.605 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:44:56.606 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_27 2022-04-10T01:44:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_27 2022-04-10T01:44:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_27 2022-04-10T01:44:56.650 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_27 2022-04-10T01:44:56.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi043.stderr:2022-04-10T01:44:56.755+0000 7f4f5dabb700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5603b918e000 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:44:56.758 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi043.stderr:2022-04-10T01:44:56.755+0000 7f4f63ac7700 -1 rbd::mirror::ImageReplayer: 0x5603b8f88400 [3/5e030d77-6c1f-4e4a-a414-67687916c7e7] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T01:44:56.881 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.4652429576343936) > weight (0.25) 2022-04-10T01:44:56.882 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.6856393006454435) > weight (0.25) 2022-04-10T01:44:56.882 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.347095542559794) > weight (0.25) 2022-04-10T01:44:56.882 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2022-04-10T01:44:56.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2022-04-10T01:44:56.883 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2022-04-10T01:44:56.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr:2022-04-10T01:44:56.882+0000 7f006e15a700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 41257) UID: 1000 2022-04-10T01:44:57.716 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-10T01:44:57.720 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:44:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_28 2022-04-10T01:44:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror image_28 snap 2022-04-10T01:44:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_28 2022-04-10T01:44:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap 2022-04-10T01:44:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:57.723 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:57.723 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:57.723 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:57.723 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/image_28@snap 2022-04-10T01:44:58.589 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:44:58.600 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_28 2022-04-10T01:44:58.600 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:58.601 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:58.601 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_28 2022-04-10T01:44:58.601 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:58.601 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:58.602 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:58.602 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:58.602 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:44:58.602 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:44:58.604 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_28 2022-04-10T01:44:58.604 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:58.604 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:58.605 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_28 2022-04-10T01:44:58.605 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:58.605 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:58.605 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:58.605 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:58.606 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_28 2022-04-10T01:44:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_28 2022-04-10T01:44:58.763 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr:2022-04-10T01:44:58.760+0000 7f9ecbff4700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f1a10b1d40 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:44:58.763 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr:2022-04-10T01:44:58.760+0000 7f9ed2801700 -1 rbd::mirror::ImageReplayer: 0x55f1a0b29800 [3/e100174d-382f-42fc-9a30-2ef4065b1b6a] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T01:44:58.884 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2022-04-10T01:44:58.885 INFO:teuthology.orchestra.run:waiting for 600 2022-04-10T01:44:59.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr:daemon-helper: command failed with exit status 1 2022-04-10T01:44:59.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-10T01:44:59.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:59.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:44:59.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_29 2022-04-10T01:44:59.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror image_29 snap 2022-04-10T01:44:59.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:59.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:44:59.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_29 2022-04-10T01:44:59.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap 2022-04-10T01:44:59.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:44:59.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:44:59.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:44:59.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:44:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/image_29@snap 2022-04-10T01:45:00.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:45:00.610 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_29 2022-04-10T01:45:00.610 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:00.610 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:00.610 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_29 2022-04-10T01:45:00.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:00.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:00.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:00.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:00.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:45:00.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:45:00.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_29 2022-04-10T01:45:00.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:00.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:00.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_29 2022-04-10T01:45:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:00.615 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_29 2022-04-10T01:45:00.658 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_29 2022-04-10T01:45:00.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi043.stderr:2022-04-10T01:45:00.762+0000 7fe8bb98b700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55b34a1eba00 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:45:00.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi043.stderr:2022-04-10T01:45:00.762+0000 7fe8c2198700 -1 rbd::mirror::ImageReplayer: 0x55b349e43c00 [3/617e30a5-cc17-4271-95fb-ba4832408157] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T01:45:01.765 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-10T01:45:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_30 2022-04-10T01:45:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror image_30 snap 2022-04-10T01:45:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_30 2022-04-10T01:45:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap 2022-04-10T01:45:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/image_30@snap 2022-04-10T01:45:02.607 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:45:02.619 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_30 2022-04-10T01:45:02.619 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:02.619 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:02.619 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_30 2022-04-10T01:45:02.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:02.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:02.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:02.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:02.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:45:02.621 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:45:02.621 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_30 2022-04-10T01:45:02.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:02.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:02.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_30 2022-04-10T01:45:02.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:02.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:02.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:02.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:02.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_30 2022-04-10T01:45:02.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_30 2022-04-10T01:45:02.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi043.stderr:2022-04-10T01:45:02.776+0000 7fe8bb98b700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55b349e9c680 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:45:02.779 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi043.stderr:2022-04-10T01:45:02.776+0000 7fe8c2198700 -1 rbd::mirror::ImageReplayer: 0x55b349fe4800 [3/e756a8db-50ef-4000-ba87-3241379f1d82] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T01:45:03.743 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-10T01:45:03.747 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:03.747 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:03.748 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_31 2022-04-10T01:45:03.748 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror image_31 snap 2022-04-10T01:45:03.748 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:03.748 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:03.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_31 2022-04-10T01:45:03.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap 2022-04-10T01:45:03.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:03.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:03.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:03.750 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:03.750 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/image_31@snap 2022-04-10T01:45:04.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:45:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_31 2022-04-10T01:45:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:04.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_31 2022-04-10T01:45:04.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:04.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:04.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:45:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:45:04.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_31 2022-04-10T01:45:04.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:04.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:04.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_31 2022-04-10T01:45:04.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:04.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:04.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:04.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:04.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_31 2022-04-10T01:45:04.619 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_31 2022-04-10T01:45:04.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi043.stderr:2022-04-10T01:45:04.722+0000 7fc2c214b700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55b0c0555380 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:45:04.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi043.stderr:2022-04-10T01:45:04.722+0000 7fc2c9159700 -1 rbd::mirror::ImageReplayer: 0x55b0c00e5e00 [3/9c9c599b-a61d-4229-a0b1-ba391a8116f1] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T01:45:04.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-04-10T01:45:04.986 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2022-04-10T01:45:04.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2022-04-10T01:45:04.987 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2022-04-10T01:45:04.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2022-04-10T01:45:04.990 INFO:tasks.rbd_mirror_thrash:waiting for 96 secs before thrashing 2022-04-10T01:45:05.733 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-10T01:45:05.737 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:05.737 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:05.738 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_32 2022-04-10T01:45:05.738 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror image_32 snap 2022-04-10T01:45:05.738 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:05.738 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:05.739 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_32 2022-04-10T01:45:05.739 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap 2022-04-10T01:45:05.739 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:05.739 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:05.739 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:05.740 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:05.740 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/image_32@snap 2022-04-10T01:45:06.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:45:06.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_32 2022-04-10T01:45:06.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:06.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:06.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_32 2022-04-10T01:45:06.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:06.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:45:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:45:06.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_32 2022-04-10T01:45:06.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:06.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:06.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_32 2022-04-10T01:45:06.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:06.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:06.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:06.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:06.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_32 2022-04-10T01:45:06.681 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_32 2022-04-10T01:45:06.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi043.stderr:2022-04-10T01:45:06.797+0000 7fe8bc18c700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55b349e9cea0 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:45:06.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi043.stderr:2022-04-10T01:45:06.797+0000 7fe8c2198700 -1 rbd::mirror::ImageReplayer: 0x55b349fe4400 [3/34f68cc3-808d-4b7f-a2e3-415ae64356a9] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T01:45:07.757 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-10T01:45:07.761 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:07.761 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:07.761 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_33 2022-04-10T01:45:07.762 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror image_33 snap 2022-04-10T01:45:07.762 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:07.762 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:07.762 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_33 2022-04-10T01:45:07.763 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap 2022-04-10T01:45:07.763 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:07.763 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:07.764 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:07.764 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:07.764 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/image_33@snap 2022-04-10T01:45:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:45:08.631 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_33 2022-04-10T01:45:08.631 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:08.631 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:08.632 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_33 2022-04-10T01:45:08.632 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:08.632 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:08.632 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:08.632 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:08.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:45:08.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:45:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_33 2022-04-10T01:45:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_33 2022-04-10T01:45:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:08.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_33 2022-04-10T01:45:08.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_33 2022-04-10T01:45:08.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi043.stderr:2022-04-10T01:45:08.750+0000 7fd7c6213700 -1 rbd::mirror::ImageMap: 0x55c162085e60 handle_peer_ack_remove: failed to remove global_image_id=3334b7aa-6174-45c6-b4b8-608b7f09db2a 2022-04-10T01:45:09.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-10T01:45:09.699 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_34 2022-04-10T01:45:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror image_34 snap 2022-04-10T01:45:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:09.701 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_34 2022-04-10T01:45:09.701 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap 2022-04-10T01:45:09.701 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:09.701 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:09.702 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:09.702 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:09.702 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/image_34@snap 2022-04-10T01:45:10.627 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:45:10.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_34 2022-04-10T01:45:10.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:10.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:10.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_34 2022-04-10T01:45:10.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:10.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:10.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:10.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:10.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:45:10.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:45:10.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_34 2022-04-10T01:45:10.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:10.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:10.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_34 2022-04-10T01:45:10.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:10.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:10.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:10.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:10.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_34 2022-04-10T01:45:10.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_34 2022-04-10T01:45:10.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi043.stderr:2022-04-10T01:45:10.745+0000 7fd7c6213700 -1 rbd::mirror::ImageMap: 0x55c162085e60 handle_peer_ack_remove: failed to remove global_image_id=fc8271db-fe00-47f6-a840-9d97ed18fbb7 2022-04-10T01:45:11.762 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-10T01:45:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_35 2022-04-10T01:45:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror image_35 snap 2022-04-10T01:45:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_35 2022-04-10T01:45:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap 2022-04-10T01:45:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/image_35@snap 2022-04-10T01:45:12.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:45:12.649 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_35 2022-04-10T01:45:12.650 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:12.650 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:12.650 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_35 2022-04-10T01:45:12.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:12.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:12.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:12.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:12.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:45:12.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:45:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_35 2022-04-10T01:45:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:12.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_35 2022-04-10T01:45:12.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:12.662 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:12.662 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:12.662 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:12.663 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_35 2022-04-10T01:45:12.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_35 2022-04-10T01:45:12.805 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi043.stderr:2022-04-10T01:45:12.804+0000 7f4f5d2ba700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5603b918e1a0 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:45:12.806 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi043.stderr:2022-04-10T01:45:12.804+0000 7f4f63ac7700 -1 rbd::mirror::ImageReplayer: 0x5603b8f88000 [3/2864f9d0-8af5-4721-a606-e52b47ecb56e] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T01:45:13.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-10T01:45:13.773 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:13.773 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:13.774 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_36 2022-04-10T01:45:13.774 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror image_36 snap 2022-04-10T01:45:13.775 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:13.775 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:13.776 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_36 2022-04-10T01:45:13.776 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap 2022-04-10T01:45:13.776 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:13.777 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:13.777 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:13.778 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:13.778 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/image_36@snap 2022-04-10T01:45:14.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:45:14.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_36 2022-04-10T01:45:14.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:14.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:14.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_36 2022-04-10T01:45:14.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:45:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:45:14.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_36 2022-04-10T01:45:14.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:14.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:14.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_36 2022-04-10T01:45:14.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:14.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:14.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:14.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:14.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_36 2022-04-10T01:45:14.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_36 2022-04-10T01:45:14.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi043.stderr:2022-04-10T01:45:14.742+0000 7fe8bb98b700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55b349e9c820 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:45:14.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi043.stderr:2022-04-10T01:45:14.749+0000 7fe8bb98b700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55b347978d00 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:45:14.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi043.stderr:2022-04-10T01:45:14.749+0000 7fe8c2999700 -1 rbd::mirror::ImageReplayer: 0x55b349fe4200 [3/347e3669-52d8-4e7d-9693-a1920cf47072] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T01:45:15.771 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-10T01:45:15.775 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:15.776 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:15.776 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_37 2022-04-10T01:45:15.777 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror image_37 snap 2022-04-10T01:45:15.777 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_37 2022-04-10T01:45:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap 2022-04-10T01:45:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:15.780 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/image_37@snap 2022-04-10T01:45:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:45:16.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_37 2022-04-10T01:45:16.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:16.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:16.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_37 2022-04-10T01:45:16.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:16.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:16.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:16.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:45:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:45:16.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_37 2022-04-10T01:45:16.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_37 2022-04-10T01:45:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_37 2022-04-10T01:45:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_37 2022-04-10T01:45:16.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi043.stderr:2022-04-10T01:45:16.768+0000 7fd7c5a12700 -1 rbd::mirror::ImageMap: 0x55c162085e60 handle_peer_ack_remove: failed to remove global_image_id=6e506c1e-5b46-476c-afb6-a70d5f9862e5 2022-04-10T01:45:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-10T01:45:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_38 2022-04-10T01:45:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror image_38 snap 2022-04-10T01:45:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_38 2022-04-10T01:45:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap 2022-04-10T01:45:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/image_38@snap 2022-04-10T01:45:18.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:45:18.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_38 2022-04-10T01:45:18.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:18.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:18.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_38 2022-04-10T01:45:18.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:18.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:18.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:18.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:18.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:45:18.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:45:18.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_38 2022-04-10T01:45:18.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:18.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_38 2022-04-10T01:45:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:18.677 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_38 2022-04-10T01:45:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_38 2022-04-10T01:45:18.826 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi043.stderr:2022-04-10T01:45:18.825+0000 7fc2c214b700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55b0c0555520 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:45:18.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi043.stderr:2022-04-10T01:45:18.833+0000 7fc2c214b700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55b0c05549c0 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:45:18.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi043.stderr:2022-04-10T01:45:18.833+0000 7fc2c8958700 -1 rbd::mirror::ImageReplayer: 0x55b0bfe57400 [3/643d8009-5a9c-482d-a7cf-86349b296923] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T01:45:19.700 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-10T01:45:19.703 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:19.704 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:19.704 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_39 2022-04-10T01:45:19.704 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror image_39 snap 2022-04-10T01:45:19.704 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:19.705 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:19.705 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_39 2022-04-10T01:45:19.705 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap 2022-04-10T01:45:19.705 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/image_39@snap 2022-04-10T01:45:20.662 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:45:20.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_39 2022-04-10T01:45:20.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:20.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:20.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_39 2022-04-10T01:45:20.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:20.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:20.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:45:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:45:20.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_39 2022-04-10T01:45:20.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:20.677 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:20.677 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_39 2022-04-10T01:45:20.677 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:20.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:20.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:20.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:20.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_39 2022-04-10T01:45:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_39 2022-04-10T01:45:20.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr:2022-04-10T01:45:20.831+0000 7f9ecc7f5700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f1a10a01a0 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:45:20.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr:2022-04-10T01:45:20.831+0000 7f9ed2801700 -1 rbd::mirror::ImageReplayer: 0x55f1a0b2ba00 [3/e2bbf2e7-46b6-4dd8-b6e1-9dea8cf3b0b4] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T01:45:21.386 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr:2022-04-10T01:45:21.384+0000 7f9ecbff4700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f1a10a1ba0 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:45:21.386 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr:2022-04-10T01:45:21.384+0000 7f9ed3002700 -1 rbd::mirror::ImageReplayer: 0x55f1a0b2ba00 [3/e2bbf2e7-46b6-4dd8-b6e1-9dea8cf3b0b4] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T01:45:21.792 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-10T01:45:21.796 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:21.797 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:21.797 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_40 2022-04-10T01:45:21.797 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror image_40 snap 2022-04-10T01:45:21.797 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:21.797 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:21.798 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_40 2022-04-10T01:45:21.798 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap 2022-04-10T01:45:21.798 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:21.798 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/image_40@snap 2022-04-10T01:45:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:45:22.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_40 2022-04-10T01:45:22.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:22.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:22.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_40 2022-04-10T01:45:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:22.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:45:22.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:45:22.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_40 2022-04-10T01:45:22.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:22.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:22.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_40 2022-04-10T01:45:22.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:22.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:22.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_40 2022-04-10T01:45:22.736 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_40 2022-04-10T01:45:22.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi043.stderr:2022-04-10T01:45:22.846+0000 7f4f5d2ba700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5603b75e5a00 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:45:22.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi043.stderr:2022-04-10T01:45:22.846+0000 7f4f642c8700 -1 rbd::mirror::ImageReplayer: 0x5603b8df3800 [3/778e4963-897c-4526-9fc0-0a90b3039381] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T01:45:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-10T01:45:23.812 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:23.813 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:23.813 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_41 2022-04-10T01:45:23.813 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror image_41 snap 2022-04-10T01:45:23.813 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:23.813 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:23.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_41 2022-04-10T01:45:23.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap 2022-04-10T01:45:23.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:23.814 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:23.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:23.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:23.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/image_41@snap 2022-04-10T01:45:24.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:45:24.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_41 2022-04-10T01:45:24.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:24.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:24.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_41 2022-04-10T01:45:24.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:24.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:24.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:24.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:24.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:45:24.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:45:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_41 2022-04-10T01:45:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_41 2022-04-10T01:45:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:24.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_41 2022-04-10T01:45:24.631 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_41 2022-04-10T01:45:24.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi043.stderr:2022-04-10T01:45:24.694+0000 7fd7c5a12700 -1 rbd::mirror::ImageMap: 0x55c162085e60 handle_peer_ack_remove: failed to remove global_image_id=af549731-3fee-45da-9830-6e51d5343406 2022-04-10T01:45:26.023 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-10T01:45:26.027 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:26.027 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:26.028 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_42 2022-04-10T01:45:26.028 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror image_42 snap 2022-04-10T01:45:26.028 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:26.028 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:26.028 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_42 2022-04-10T01:45:26.029 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap 2022-04-10T01:45:26.029 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:26.029 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:26.029 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:26.030 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:26.030 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/image_42@snap 2022-04-10T01:45:26.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:45:26.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_42 2022-04-10T01:45:26.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:26.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:26.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_42 2022-04-10T01:45:26.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:26.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:26.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:26.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:45:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:45:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_42 2022-04-10T01:45:26.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:26.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:26.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_42 2022-04-10T01:45:26.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:26.698 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:26.698 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:26.698 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:26.698 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_42 2022-04-10T01:45:26.948 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_42 2022-04-10T01:45:27.018 INFO:tasks.ceph.cluster2.osd.2.smithi043.stderr:2022-04-10T01:45:27.016+0000 7f8be0453700 -1 reset not still connected to 0x55f2b7534750 2022-04-10T01:45:27.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi043.stderr:2022-04-10T01:45:27.060+0000 7fc2c214b700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55b0c0541380 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:45:27.063 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi043.stderr:2022-04-10T01:45:27.060+0000 7fc2c8958700 -1 rbd::mirror::ImageReplayer: 0x55b0c00e5c00 [3/ceed605c-dff2-49f6-bdf7-f3ed1ce8dfcd] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T01:45:27.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-10T01:45:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_43 2022-04-10T01:45:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror image_43 snap 2022-04-10T01:45:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_43 2022-04-10T01:45:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap 2022-04-10T01:45:27.823 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:27.823 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:27.823 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:27.823 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:27.824 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/image_43@snap 2022-04-10T01:45:28.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:45:28.707 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_43 2022-04-10T01:45:28.707 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:28.707 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:28.707 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_43 2022-04-10T01:45:28.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:28.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:28.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:28.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:28.709 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:45:28.709 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:45:28.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_43 2022-04-10T01:45:28.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:28.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:28.712 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_43 2022-04-10T01:45:28.712 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:28.712 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_43 2022-04-10T01:45:28.753 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_43 2022-04-10T01:45:28.868 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr:2022-04-10T01:45:28.866+0000 7f9ecbff4700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f1a10a1d40 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:45:28.869 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr:2022-04-10T01:45:28.866+0000 7f9ed3002700 -1 rbd::mirror::ImageReplayer: 0x55f1a0b29400 [3/bcd924f1-5832-47a7-92fc-1dbfdf17dc65] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T01:45:29.702 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-10T01:45:29.706 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:29.706 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_44 2022-04-10T01:45:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror image_44 snap 2022-04-10T01:45:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_44 2022-04-10T01:45:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap 2022-04-10T01:45:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/image_44@snap 2022-04-10T01:45:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:45:30.707 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_44 2022-04-10T01:45:30.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:30.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:30.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_44 2022-04-10T01:45:30.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:30.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:30.709 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:30.709 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:30.709 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:45:30.709 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:45:30.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_44 2022-04-10T01:45:30.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:30.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:30.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_44 2022-04-10T01:45:30.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:30.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:30.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:30.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:30.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_44 2022-04-10T01:45:30.753 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_44 2022-04-10T01:45:30.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi043.stderr:2022-04-10T01:45:30.871+0000 7f4f5d2ba700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5603b918e4e0 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:45:30.874 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi043.stderr:2022-04-10T01:45:30.871+0000 7f4f63ac7700 -1 rbd::mirror::ImageReplayer: 0x5603b8df3000 [3/06cfc5f2-3677-4f04-83bb-d75fb95a469d] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T01:45:31.830 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-10T01:45:31.834 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:31.834 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:31.834 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_45 2022-04-10T01:45:31.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror image_45 snap 2022-04-10T01:45:31.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:31.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:31.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_45 2022-04-10T01:45:31.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap 2022-04-10T01:45:31.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:31.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:31.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:31.837 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:31.837 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/image_45@snap 2022-04-10T01:45:32.699 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:45:32.714 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_45 2022-04-10T01:45:32.715 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:32.715 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:32.716 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_45 2022-04-10T01:45:32.716 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:32.716 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:32.717 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:32.717 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:32.718 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:45:32.718 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:45:32.719 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_45 2022-04-10T01:45:32.719 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:32.720 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:32.720 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_45 2022-04-10T01:45:32.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:32.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:32.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:32.722 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:32.722 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_45 2022-04-10T01:45:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_45 2022-04-10T01:45:32.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi043.stderr:2022-04-10T01:45:32.877+0000 7fc2c294c700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55b0c05416c0 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:45:32.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi043.stderr:2022-04-10T01:45:32.877+0000 7fc2c8958700 -1 rbd::mirror::ImageReplayer: 0x55b0c0273a00 [3/eb158ebf-3a87-4b1d-a50e-a8e02e82428f] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T01:45:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-10T01:45:34.016 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:34.016 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:34.016 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_46 2022-04-10T01:45:34.017 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror image_46 snap 2022-04-10T01:45:34.017 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:34.017 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:34.017 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_46 2022-04-10T01:45:34.018 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap 2022-04-10T01:45:34.018 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:34.018 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:34.018 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:34.018 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/image_46@snap 2022-04-10T01:45:34.567 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:45:34.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_46 2022-04-10T01:45:34.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:34.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_46 2022-04-10T01:45:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:45:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:45:34.580 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_46 2022-04-10T01:45:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_46 2022-04-10T01:45:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_46 2022-04-10T01:45:34.827 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_46 2022-04-10T01:45:34.937 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi043.stderr:2022-04-10T01:45:34.934+0000 7fc2c214b700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55b0c0541860 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:45:34.937 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi043.stderr:2022-04-10T01:45:34.934+0000 7fc2c8958700 -1 rbd::mirror::ImageReplayer: 0x55b0c00e5a00 [3/c83a7972-17a4-4b3c-b33c-85a8de6039e5] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T01:45:35.841 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-10T01:45:35.845 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:35.845 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:35.846 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_47 2022-04-10T01:45:35.846 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror image_47 snap 2022-04-10T01:45:35.846 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:35.846 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:35.847 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_47 2022-04-10T01:45:35.847 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap 2022-04-10T01:45:35.847 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:35.847 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:35.847 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/image_47@snap 2022-04-10T01:45:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:45:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_47 2022-04-10T01:45:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:36.723 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:36.723 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_47 2022-04-10T01:45:36.724 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:36.724 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:36.725 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:36.725 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:36.725 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:45:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:45:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_47 2022-04-10T01:45:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_47 2022-04-10T01:45:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:36.729 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:36.729 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:36.729 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:36.729 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_47 2022-04-10T01:45:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_47 2022-04-10T01:45:36.890 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi043.stderr:2022-04-10T01:45:36.887+0000 7fc2c214b700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55b0c0541a00 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:45:36.890 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi043.stderr:2022-04-10T01:45:36.887+0000 7fc2c8958700 -1 rbd::mirror::ImageReplayer: 0x55b0bc11fa00 [3/a7fe8d55-51fd-4578-8dbb-89aefafe026d] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T01:45:37.848 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-10T01:45:37.851 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:37.852 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:37.852 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_48 2022-04-10T01:45:37.852 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror image_48 snap 2022-04-10T01:45:37.852 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_48 2022-04-10T01:45:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap 2022-04-10T01:45:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:37.854 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:37.854 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:37.854 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/image_48@snap 2022-04-10T01:45:38.716 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:45:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_48 2022-04-10T01:45:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:38.729 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_48 2022-04-10T01:45:38.729 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:38.729 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:38.730 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:38.730 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:38.730 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:45:38.730 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:45:38.731 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_48 2022-04-10T01:45:38.731 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:38.732 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:38.732 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_48 2022-04-10T01:45:38.732 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:38.732 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:38.732 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:38.733 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:38.733 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_48 2022-04-10T01:45:38.782 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_48 2022-04-10T01:45:38.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi043.stderr:2022-04-10T01:45:38.992+0000 7fe8bb98b700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55b347944000 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:45:38.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi043.stderr:2022-04-10T01:45:38.992+0000 7fe8c2999700 -1 rbd::mirror::ImageReplayer: 0x55b349e43a00 [3/dc4e02d6-0aa7-4947-95ef-025b13fdc64d] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T01:45:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-10T01:45:39.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:39.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:39.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_49 2022-04-10T01:45:39.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror image_49 snap 2022-04-10T01:45:39.709 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:39.709 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:39.709 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_49 2022-04-10T01:45:39.709 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap 2022-04-10T01:45:39.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:39.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:39.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:39.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:39.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/image_49@snap 2022-04-10T01:45:40.720 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:45:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_49 2022-04-10T01:45:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_49 2022-04-10T01:45:40.731 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:40.731 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:40.731 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:40.731 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:40.732 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:45:40.732 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:45:40.733 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_49 2022-04-10T01:45:40.734 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:40.734 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:40.734 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_49 2022-04-10T01:45:40.735 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:40.735 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:40.735 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_49 2022-04-10T01:45:40.783 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_49 2022-04-10T01:45:40.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi043.stderr:2022-04-10T01:45:40.905+0000 7fe8bc18c700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55b34a26fa00 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:45:40.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi043.stderr:2022-04-10T01:45:40.905+0000 7fe8c2999700 -1 rbd::mirror::ImageReplayer: 0x55b349fe4000 [3/dadd4ce2-bc41-4878-8215-b7901a7d9b7c] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T01:45:41.855 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-10T01:45:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_50 2022-04-10T01:45:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_snapshot cluster2 mirror image_50 snap 2022-04-10T01:45:41.861 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:41.861 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:41.861 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_50 2022-04-10T01:45:41.861 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local snap=snap 2022-04-10T01:45:41.862 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:41.862 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:41.862 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap rm mirror/image_50@snap 2022-04-10T01:45:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing snap: 100% complete...done. 2022-04-10T01:45:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_50 2022-04-10T01:45:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_50 2022-04-10T01:45:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:45:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:45:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_50 2022-04-10T01:45:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_50 2022-04-10T01:45:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:42.754 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:42.754 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:42.754 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:42.754 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_50 2022-04-10T01:45:42.796 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_50 2022-04-10T01:45:42.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi043.stderr:2022-04-10T01:45:42.903+0000 7f4f5d2ba700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5603b75e51e0 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:45:42.906 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi043.stderr:2022-04-10T01:45:42.903+0000 7f4f642c8700 -1 rbd::mirror::ImageReplayer: 0x5603b8f88600 [3/a17938e9-2a1e-432d-9eca-f1aacf780959] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T01:45:43.869 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2022-04-10T01:45:43.872 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:43.872 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ testlog 'TEST: image deletions should propagate' 2022-04-10T01:45:43.872 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ tee -a /tmp/tmp.rgwz74WspX/rbd-mirror.test.log 2022-04-10T01:45:43.873 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ date '+%F %T' 2022-04-10T01:45:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo 2022-04-10 01:45:43 TEST: image deletions should propagate 2022-04-10T01:45:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:2022-04-10 01:45:43 TEST: image deletions should propagate 2022-04-10T01:45:43.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_pool_images cluster1 mirror 0 2022-04-10T01:45:43.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:43.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:43.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_count=0 2022-04-10T01:45:43.876 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:43.876 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local count 2022-04-10T01:45:43.876 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local last_count=0 2022-04-10T01:45:43.876 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ true 2022-04-10T01:45:43.876 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ seq 1 40 2022-04-10T01:45:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in `seq 1 40` 2022-04-10T01:45:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 1 -ne 1 2022-04-10T01:45:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2022-04-10T01:45:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep 'images: ' 2022-04-10T01:45:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cut '-d ' -f 2 2022-04-10T01:45:43.905 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: failed to query services: (13) Permission denied 2022-04-10T01:45:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ count=0 2022-04-10T01:45:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 0 = 0 2022-04-10T01:45:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_pool_healthy cluster1 mirror 0 2022-04-10T01:45:43.909 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:43.909 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:43.909 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:43.909 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state 2022-04-10T01:45:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ seq 1 40 2022-04-10T01:45:43.911 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in `seq 1 40` 2022-04-10T01:45:43.911 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test 1 -ne 1 2022-04-10T01:45:43.911 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2022-04-10T01:45:43.911 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ grep 'image health:' 2022-04-10T01:45:43.912 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cut '-d ' -f 3 2022-04-10T01:45:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: failed to query services: (13) Permission denied 2022-04-10T01:45:43.941 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ state=OK 2022-04-10T01:45:43.941 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test OK = ERROR 2022-04-10T01:45:43.941 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test OK = OK 2022-04-10T01:45:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ seq 1 50 2022-04-10T01:45:43.944 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_1 2022-04-10T01:45:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2022-04-10T01:45:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:43.946 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_1 2022-04-10T01:45:43.946 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:45:43.946 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:43.946 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:43.947 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:43.947 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:43.947 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:43.947 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:43.947 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_1 2022-04-10T01:45:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_1 2022-04-10T01:45:43.949 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:43.949 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:43.949 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:43.949 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:43.949 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2022-04-10T01:45:43.950 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:43.971 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_1: (2) No such file or directory 2022-04-10T01:45:43.974 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:45:43.974 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:45:43.974 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:45:44.076 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2022-04-10T01:45:44.077 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:44.077 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:44.077 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_1 2022-04-10T01:45:44.078 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:45:44.078 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:44.078 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:45:44.078 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:45:44.078 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:44.079 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:44.079 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:44.079 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:44.079 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_1 2022-04-10T01:45:44.080 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:44.080 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:44.080 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_1 2022-04-10T01:45:44.080 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:44.081 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:44.081 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:44.081 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:44.081 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2022-04-10T01:45:44.081 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:44.103 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_1: (2) No such file or directory 2022-04-10T01:45:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:45:44.109 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:44.109 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:45:44.109 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:44.109 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:44.110 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_2 2022-04-10T01:45:44.110 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2022-04-10T01:45:44.110 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:44.110 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:44.110 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_2 2022-04-10T01:45:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:45:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:44.112 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:44.112 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:44.112 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:44.112 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:44.112 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_2 2022-04-10T01:45:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_2 2022-04-10T01:45:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2022-04-10T01:45:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:44.136 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_2: (2) No such file or directory 2022-04-10T01:45:44.139 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:45:44.139 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:45:44.139 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:45:44.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2022-04-10T01:45:44.242 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:44.242 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:44.242 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_2 2022-04-10T01:45:44.242 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:45:44.243 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:44.243 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:45:44.243 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:45:44.243 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:44.243 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:44.244 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:44.244 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:44.244 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_2 2022-04-10T01:45:44.244 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:44.244 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:44.245 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_2 2022-04-10T01:45:44.245 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:44.245 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:44.245 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:44.246 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:44.246 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2022-04-10T01:45:44.246 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:44.270 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_2: (2) No such file or directory 2022-04-10T01:45:44.275 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:45:44.275 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:44.276 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:45:44.276 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:44.276 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:44.276 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_3 2022-04-10T01:45:44.277 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2022-04-10T01:45:44.277 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:44.277 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:44.277 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_3 2022-04-10T01:45:44.277 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:45:44.278 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:44.278 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:44.278 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:44.278 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:44.279 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:44.279 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:44.279 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:44.279 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_3 2022-04-10T01:45:44.280 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:44.280 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:44.280 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_3 2022-04-10T01:45:44.280 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:44.280 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:44.281 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:44.281 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:44.281 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2022-04-10T01:45:44.281 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:44.303 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_3: (2) No such file or directory 2022-04-10T01:45:44.306 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:45:44.306 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:45:44.306 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:45:44.408 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2022-04-10T01:45:44.409 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:44.409 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:44.409 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_3 2022-04-10T01:45:44.409 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:45:44.409 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:44.410 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:45:44.410 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:45:44.410 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:44.410 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_3 2022-04-10T01:45:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:44.412 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:44.412 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_3 2022-04-10T01:45:44.412 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:44.412 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:44.412 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:44.413 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:44.413 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2022-04-10T01:45:44.413 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:44.436 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_3: (2) No such file or directory 2022-04-10T01:45:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:45:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:45:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:44.442 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:44.442 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_4 2022-04-10T01:45:44.442 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2022-04-10T01:45:44.442 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:44.442 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:44.443 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_4 2022-04-10T01:45:44.443 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:45:44.443 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:44.443 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:44.443 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:44.444 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:44.444 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:44.444 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:44.444 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:44.444 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_4 2022-04-10T01:45:44.445 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:44.445 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:44.445 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_4 2022-04-10T01:45:44.445 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:44.445 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:44.446 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:44.446 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:44.446 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2022-04-10T01:45:44.446 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:44.468 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_4: (2) No such file or directory 2022-04-10T01:45:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:45:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:45:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:45:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2022-04-10T01:45:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:44.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_4 2022-04-10T01:45:44.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:45:44.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:44.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:45:44.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:45:44.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:44.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:44.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:44.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:44.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_4 2022-04-10T01:45:44.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:44.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:44.577 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_4 2022-04-10T01:45:44.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:44.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:44.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:44.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:44.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2022-04-10T01:45:44.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:44.602 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_4: (2) No such file or directory 2022-04-10T01:45:44.607 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:45:44.607 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:44.608 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:45:44.608 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:44.608 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:44.608 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_5 2022-04-10T01:45:44.609 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2022-04-10T01:45:44.609 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:44.609 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:44.609 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_5 2022-04-10T01:45:44.610 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:45:44.610 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:44.610 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:44.610 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:44.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:44.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:44.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:44.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:44.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_5 2022-04-10T01:45:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_5 2022-04-10T01:45:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:44.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:44.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:44.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2022-04-10T01:45:44.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_5: (2) No such file or directory 2022-04-10T01:45:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:45:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:45:44.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:45:44.740 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2022-04-10T01:45:44.740 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:44.741 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:44.741 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_5 2022-04-10T01:45:44.741 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:45:44.741 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:44.742 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:45:44.742 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:45:44.742 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:44.742 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:44.742 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:44.743 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:44.743 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_5 2022-04-10T01:45:44.743 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:44.743 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:44.744 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_5 2022-04-10T01:45:44.744 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:44.744 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:44.744 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:44.744 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:44.745 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2022-04-10T01:45:44.745 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:44.768 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_5: (2) No such file or directory 2022-04-10T01:45:44.773 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:45:44.773 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:44.773 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:45:44.773 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:44.774 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:44.774 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_6 2022-04-10T01:45:44.774 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2022-04-10T01:45:44.774 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:44.775 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:44.775 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_6 2022-04-10T01:45:44.775 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:45:44.775 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:44.775 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_6 2022-04-10T01:45:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_6 2022-04-10T01:45:44.778 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:44.778 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:44.778 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:44.778 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:44.778 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2022-04-10T01:45:44.779 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:44.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_6: (2) No such file or directory 2022-04-10T01:45:44.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:45:44.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:45:44.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:45:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2022-04-10T01:45:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:44.906 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:44.906 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_6 2022-04-10T01:45:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:45:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:45:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:45:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:44.913 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_6 2022-04-10T01:45:44.913 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:44.913 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:44.913 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_6 2022-04-10T01:45:44.914 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:44.914 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:44.914 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:44.914 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:44.914 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2022-04-10T01:45:44.915 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_6: (2) No such file or directory 2022-04-10T01:45:44.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:45:44.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:44.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:45:44.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:44.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:44.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_7 2022-04-10T01:45:44.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2022-04-10T01:45:44.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_7 2022-04-10T01:45:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:45:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_7 2022-04-10T01:45:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_7 2022-04-10T01:45:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:44.944 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2022-04-10T01:45:44.944 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:44.966 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_7: (2) No such file or directory 2022-04-10T01:45:44.969 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:45:44.969 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:45:44.970 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:45:45.072 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2022-04-10T01:45:45.072 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:45.072 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:45.072 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_7 2022-04-10T01:45:45.073 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:45:45.073 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:45.073 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:45:45.073 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:45:45.074 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:45.074 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:45.074 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:45.074 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:45.074 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_7 2022-04-10T01:45:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_7 2022-04-10T01:45:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:45.076 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:45.076 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:45.076 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:45.076 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2022-04-10T01:45:45.076 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:45.099 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_7: (2) No such file or directory 2022-04-10T01:45:45.104 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:45:45.104 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:45.105 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:45:45.105 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:45.105 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:45.105 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_8 2022-04-10T01:45:45.106 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2022-04-10T01:45:45.106 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:45.106 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:45.106 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_8 2022-04-10T01:45:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:45:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:45.108 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:45.108 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:45.108 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:45.108 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_8 2022-04-10T01:45:45.108 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:45.109 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:45.109 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_8 2022-04-10T01:45:45.109 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:45.109 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:45.110 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:45.110 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:45.110 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2022-04-10T01:45:45.110 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:45.131 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_8: (2) No such file or directory 2022-04-10T01:45:45.134 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:45:45.134 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:45:45.135 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:45:45.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2022-04-10T01:45:45.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:45.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:45.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_8 2022-04-10T01:45:45.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:45:45.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:45.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:45:45.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:45:45.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:45.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:45.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:45.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:45.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_8 2022-04-10T01:45:45.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:45.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:45.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_8 2022-04-10T01:45:45.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:45.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:45.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:45.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:45.241 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2022-04-10T01:45:45.242 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:45.265 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_8: (2) No such file or directory 2022-04-10T01:45:45.270 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:45:45.270 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:45.270 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:45:45.271 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:45.271 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:45.271 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_9 2022-04-10T01:45:45.271 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2022-04-10T01:45:45.271 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:45.272 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:45.272 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_9 2022-04-10T01:45:45.272 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:45:45.272 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:45.273 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:45.273 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:45.273 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:45.273 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:45.273 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:45.274 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:45.274 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_9 2022-04-10T01:45:45.274 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:45.274 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:45.274 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_9 2022-04-10T01:45:45.275 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:45.275 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:45.275 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:45.275 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:45.275 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2022-04-10T01:45:45.276 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:45.297 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_9: (2) No such file or directory 2022-04-10T01:45:45.300 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:45:45.300 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:45:45.300 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:45:45.403 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2022-04-10T01:45:45.404 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:45.404 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:45.404 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_9 2022-04-10T01:45:45.404 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:45:45.405 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:45.405 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:45:45.405 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:45:45.405 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:45.405 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:45.406 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:45.406 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:45.406 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_9 2022-04-10T01:45:45.407 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:45.407 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:45.407 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_9 2022-04-10T01:45:45.407 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:45.408 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:45.408 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:45.408 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:45.408 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2022-04-10T01:45:45.409 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:45.431 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_9: (2) No such file or directory 2022-04-10T01:45:45.436 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:45:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:45:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_10 2022-04-10T01:45:45.438 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2022-04-10T01:45:45.438 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:45.438 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:45.438 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_10 2022-04-10T01:45:45.439 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:45:45.439 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:45.439 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:45.439 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:45.439 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:45.440 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:45.440 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:45.440 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:45.440 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_10 2022-04-10T01:45:45.441 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:45.441 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:45.441 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_10 2022-04-10T01:45:45.441 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:45.441 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:45.442 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:45.442 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:45.442 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2022-04-10T01:45:45.442 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_10: (2) No such file or directory 2022-04-10T01:45:45.466 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:45:45.466 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:45:45.467 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:45:45.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2022-04-10T01:45:45.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:45.569 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_10 2022-04-10T01:45:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:45:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:45:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:45:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_10 2022-04-10T01:45:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_10 2022-04-10T01:45:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2022-04-10T01:45:45.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:45.596 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_10: (2) No such file or directory 2022-04-10T01:45:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:45:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:45:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_11 2022-04-10T01:45:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_11 deleted 2022-04-10T01:45:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_11 2022-04-10T01:45:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:45:45.604 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:45.604 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:45.604 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:45.604 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:45.604 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:45.605 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:45.605 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:45.605 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_11 2022-04-10T01:45:45.605 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:45.606 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:45.606 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_11 2022-04-10T01:45:45.606 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:45.606 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:45.606 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:45.607 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:45.607 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2022-04-10T01:45:45.607 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:45.845 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_11: (2) No such file or directory 2022-04-10T01:45:45.848 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:45:45.848 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:45:45.849 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:45:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_11 deleted '' 2022-04-10T01:45:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_11 2022-04-10T01:45:45.952 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:45:45.952 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:45.952 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:45:45.952 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:45:45.953 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:45.953 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:45.953 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:45.953 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:45.953 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_11 2022-04-10T01:45:45.954 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:45.954 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:45.954 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_11 2022-04-10T01:45:45.954 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2022-04-10T01:45:45.956 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_11: (2) No such file or directory 2022-04-10T01:45:45.984 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:45:45.985 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:45.985 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:45:45.985 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:45.985 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_12 2022-04-10T01:45:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_12 deleted 2022-04-10T01:45:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:45.987 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_12 2022-04-10T01:45:45.987 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:45:45.987 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:45.987 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:45.988 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:45.988 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:45.988 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:45.988 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:45.988 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:45.989 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_12 2022-04-10T01:45:45.989 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:45.989 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:45.989 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_12 2022-04-10T01:45:45.990 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:45.990 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:45.990 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:45.990 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:45.990 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2022-04-10T01:45:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:46.012 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_12: (2) No such file or directory 2022-04-10T01:45:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:45:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:45:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:45:46.117 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_12 deleted '' 2022-04-10T01:45:46.118 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:46.118 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:46.118 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_12 2022-04-10T01:45:46.118 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:45:46.119 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:46.119 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:45:46.119 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:45:46.119 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:46.119 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:46.120 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:46.120 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:46.120 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_12 2022-04-10T01:45:46.120 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:46.121 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:46.121 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_12 2022-04-10T01:45:46.121 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:46.121 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:46.121 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:46.122 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:46.122 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2022-04-10T01:45:46.122 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:46.347 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_12: (2) No such file or directory 2022-04-10T01:45:46.352 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:45:46.352 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:46.353 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:45:46.353 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:46.353 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_13 2022-04-10T01:45:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_13 deleted 2022-04-10T01:45:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:46.355 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_13 2022-04-10T01:45:46.355 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:45:46.355 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:46.355 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:46.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:46.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:46.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:46.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:46.357 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:46.357 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_13 2022-04-10T01:45:46.357 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:46.357 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:46.357 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_13 2022-04-10T01:45:46.358 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:46.358 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:46.358 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:46.358 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:46.359 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2022-04-10T01:45:46.359 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:46.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_13: (2) No such file or directory 2022-04-10T01:45:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:45:46.383 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:45:46.383 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:45:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_13 deleted '' 2022-04-10T01:45:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:46.486 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_13 2022-04-10T01:45:46.486 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:45:46.486 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:46.486 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:45:46.487 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:45:46.487 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:46.487 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:46.487 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:46.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:46.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_13 2022-04-10T01:45:46.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:46.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:46.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_13 2022-04-10T01:45:46.489 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:46.489 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:46.489 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:46.489 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:46.489 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2022-04-10T01:45:46.490 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:46.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_13: (2) No such file or directory 2022-04-10T01:45:46.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:45:46.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:46.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:45:46.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:46.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:46.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_14 2022-04-10T01:45:46.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_14 deleted 2022-04-10T01:45:46.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:46.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:46.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_14 2022-04-10T01:45:46.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:45:46.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:46.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:46.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:46.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:46.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:46.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:46.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:46.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_14 2022-04-10T01:45:46.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:46.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:46.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_14 2022-04-10T01:45:46.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:46.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:46.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:46.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:46.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2022-04-10T01:45:46.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:46.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_14: (2) No such file or directory 2022-04-10T01:45:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:45:46.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:45:46.548 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:45:46.650 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_14 deleted '' 2022-04-10T01:45:46.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:46.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:46.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_14 2022-04-10T01:45:46.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:45:46.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:46.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:45:46.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:45:46.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_14 2022-04-10T01:45:46.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:46.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:46.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_14 2022-04-10T01:45:46.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:46.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:46.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:46.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:46.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2022-04-10T01:45:46.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:46.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_14: (2) No such file or directory 2022-04-10T01:45:46.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:45:46.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:46.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:45:46.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:46.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:46.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_15 2022-04-10T01:45:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_15 deleted 2022-04-10T01:45:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_15 2022-04-10T01:45:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:45:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:46.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:46.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:46.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:46.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_15 2022-04-10T01:45:46.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:46.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:46.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_15 2022-04-10T01:45:46.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:46.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:46.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:46.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:46.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2022-04-10T01:45:46.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:46.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_15: (2) No such file or directory 2022-04-10T01:45:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:45:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:45:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:45:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_15 deleted '' 2022-04-10T01:45:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_15 2022-04-10T01:45:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:45:46.882 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:46.882 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:45:46.883 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:45:46.883 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:46.883 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:46.884 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:46.884 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:46.885 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_15 2022-04-10T01:45:46.885 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:46.885 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_15 2022-04-10T01:45:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2022-04-10T01:45:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:46.889 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_15: (2) No such file or directory 2022-04-10T01:45:46.889 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:45:46.890 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:46.890 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:45:46.890 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:46.891 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:46.891 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_16 2022-04-10T01:45:46.892 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_16 deleted 2022-04-10T01:45:46.892 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:46.892 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:46.893 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_16 2022-04-10T01:45:46.893 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:45:46.894 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:46.894 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:46.895 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:46.895 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:46.895 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:46.896 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:46.896 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:46.897 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_16 2022-04-10T01:45:46.897 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:46.898 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:46.898 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_16 2022-04-10T01:45:46.898 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:46.899 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:46.899 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:46.900 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:46.900 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2022-04-10T01:45:46.901 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:46.902 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_16: (2) No such file or directory 2022-04-10T01:45:46.903 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:45:46.903 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:45:46.903 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:45:46.994 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_16 deleted '' 2022-04-10T01:45:46.995 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:46.995 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:46.995 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_16 2022-04-10T01:45:46.995 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:45:46.995 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:46.996 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:45:46.996 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:45:46.996 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:46.996 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:46.997 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:46.997 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:46.997 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_16 2022-04-10T01:45:46.997 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:46.997 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:46.998 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_16 2022-04-10T01:45:46.998 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:46.998 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:46.998 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:46.999 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:46.999 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2022-04-10T01:45:46.999 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_16: (2) No such file or directory 2022-04-10T01:45:47.026 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:45:47.027 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:47.027 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:45:47.027 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:47.027 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_17 2022-04-10T01:45:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_17 deleted 2022-04-10T01:45:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_17 2022-04-10T01:45:47.029 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:45:47.029 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:47.029 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:47.030 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:47.030 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:47.030 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:47.030 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_17 2022-04-10T01:45:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_17 2022-04-10T01:45:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:47.033 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2022-04-10T01:45:47.033 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_17: (2) No such file or directory 2022-04-10T01:45:47.055 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:45:47.055 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:45:47.055 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:45:47.157 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_17 deleted '' 2022-04-10T01:45:47.158 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:47.158 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:47.158 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_17 2022-04-10T01:45:47.159 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:45:47.159 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:47.159 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:45:47.159 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:45:47.159 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:47.161 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_17 2022-04-10T01:45:47.161 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_17 2022-04-10T01:45:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2022-04-10T01:45:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:47.184 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_17: (2) No such file or directory 2022-04-10T01:45:47.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:45:47.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:47.189 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:45:47.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:47.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:47.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_18 2022-04-10T01:45:47.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_18 deleted 2022-04-10T01:45:47.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:47.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:47.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_18 2022-04-10T01:45:47.191 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:45:47.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:47.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:47.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:47.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:47.192 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:47.193 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:47.193 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:47.193 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_18 2022-04-10T01:45:47.193 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:47.193 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_18 2022-04-10T01:45:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:47.195 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:47.195 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2022-04-10T01:45:47.195 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:47.217 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_18: (2) No such file or directory 2022-04-10T01:45:47.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:45:47.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:45:47.220 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:45:47.322 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_18 deleted '' 2022-04-10T01:45:47.322 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_18 2022-04-10T01:45:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:45:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:45:47.324 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:45:47.324 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:47.324 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:47.324 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:47.324 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:47.325 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_18 2022-04-10T01:45:47.325 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:47.325 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:47.325 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_18 2022-04-10T01:45:47.325 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2022-04-10T01:45:47.327 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:47.350 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_18: (2) No such file or directory 2022-04-10T01:45:47.354 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:45:47.355 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:47.355 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:45:47.355 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:47.355 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:47.355 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_19 2022-04-10T01:45:47.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_19 deleted 2022-04-10T01:45:47.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:47.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:47.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_19 2022-04-10T01:45:47.357 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:45:47.357 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:47.357 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:47.357 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:47.358 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:47.358 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:47.358 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:47.358 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:47.359 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_19 2022-04-10T01:45:47.359 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:47.359 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:47.359 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_19 2022-04-10T01:45:47.360 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:47.360 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:47.360 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:47.360 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:47.360 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2022-04-10T01:45:47.361 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:47.382 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_19: (2) No such file or directory 2022-04-10T01:45:47.384 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:45:47.385 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:45:47.385 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:45:47.489 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_19 deleted '' 2022-04-10T01:45:47.490 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:47.491 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:47.491 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_19 2022-04-10T01:45:47.491 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:45:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:45:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:45:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:47.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:47.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:47.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:47.495 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_19 2022-04-10T01:45:47.495 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:47.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:47.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_19 2022-04-10T01:45:47.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:47.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:47.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:47.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:47.498 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2022-04-10T01:45:47.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:47.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_19: (2) No such file or directory 2022-04-10T01:45:47.519 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:45:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:45:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:47.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:47.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_20 2022-04-10T01:45:47.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_20 deleted 2022-04-10T01:45:47.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:47.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:47.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_20 2022-04-10T01:45:47.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:45:47.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:47.522 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:47.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:47.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:47.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:47.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:47.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:47.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_20 2022-04-10T01:45:47.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:47.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:47.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_20 2022-04-10T01:45:47.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:47.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:47.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:47.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:47.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2022-04-10T01:45:47.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:47.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_20: (2) No such file or directory 2022-04-10T01:45:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:45:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:45:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:45:47.651 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_20 deleted '' 2022-04-10T01:45:47.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:47.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:47.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_20 2022-04-10T01:45:47.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:45:47.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:47.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:45:47.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:45:47.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:47.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:47.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:47.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:47.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_20 2022-04-10T01:45:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_20 2022-04-10T01:45:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2022-04-10T01:45:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:47.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_20: (2) No such file or directory 2022-04-10T01:45:47.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:45:47.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:47.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:45:47.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:47.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:47.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_21 2022-04-10T01:45:47.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_21 deleted 2022-04-10T01:45:47.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:47.686 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_21 2022-04-10T01:45:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:45:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:47.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_21 2022-04-10T01:45:47.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:47.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:47.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_21 2022-04-10T01:45:47.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:47.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:47.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:47.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:47.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2022-04-10T01:45:47.691 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:47.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_21: (2) No such file or directory 2022-04-10T01:45:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:45:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:45:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:45:47.816 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_21 deleted '' 2022-04-10T01:45:47.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:47.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:47.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_21 2022-04-10T01:45:47.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:45:47.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:47.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:45:47.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:45:47.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:47.819 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:47.819 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:47.819 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:47.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_21 2022-04-10T01:45:47.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:47.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:47.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_21 2022-04-10T01:45:47.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:47.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:47.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:47.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:47.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2022-04-10T01:45:47.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:47.843 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_21: (2) No such file or directory 2022-04-10T01:45:47.848 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:45:47.848 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:47.848 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:45:47.848 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:47.849 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:47.849 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_22 2022-04-10T01:45:47.849 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_22 deleted 2022-04-10T01:45:47.849 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:47.849 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_22 2022-04-10T01:45:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:45:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:47.851 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:47.851 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:47.851 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:47.851 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:47.852 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:47.852 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:47.852 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_22 2022-04-10T01:45:47.852 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:47.852 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:47.853 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_22 2022-04-10T01:45:47.853 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:47.853 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:47.853 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2022-04-10T01:45:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:47.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_22: (2) No such file or directory 2022-04-10T01:45:47.882 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:45:47.883 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:45:47.883 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:45:47.980 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_22 deleted '' 2022-04-10T01:45:47.980 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:47.981 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:47.981 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_22 2022-04-10T01:45:47.981 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:45:47.981 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:47.981 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:45:47.982 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:45:47.982 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:47.982 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:47.982 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:47.983 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:47.983 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_22 2022-04-10T01:45:47.983 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:47.983 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:47.984 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_22 2022-04-10T01:45:47.984 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:47.984 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:47.984 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:47.985 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:47.985 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2022-04-10T01:45:47.985 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:48.007 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_22: (2) No such file or directory 2022-04-10T01:45:48.012 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:45:48.012 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:48.013 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:45:48.013 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:48.013 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:48.013 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_23 2022-04-10T01:45:48.014 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_23 deleted 2022-04-10T01:45:48.014 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:48.014 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:48.014 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_23 2022-04-10T01:45:48.015 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:45:48.015 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:48.015 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:48.015 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:48.016 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:48.016 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:48.016 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:48.016 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:48.017 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_23 2022-04-10T01:45:48.017 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:48.017 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:48.017 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_23 2022-04-10T01:45:48.018 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:48.018 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:48.018 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:48.018 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:48.019 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2022-04-10T01:45:48.019 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:48.040 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_23: (2) No such file or directory 2022-04-10T01:45:48.047 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:45:48.047 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:45:48.047 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:45:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_23 deleted '' 2022-04-10T01:45:48.146 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:48.146 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:48.146 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_23 2022-04-10T01:45:48.146 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:45:48.147 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:48.147 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:45:48.147 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:45:48.147 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:48.147 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:48.148 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:48.148 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:48.148 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_23 2022-04-10T01:45:48.148 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_23 2022-04-10T01:45:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:48.150 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:48.150 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2022-04-10T01:45:48.150 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:48.173 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_23: (2) No such file or directory 2022-04-10T01:45:48.178 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:45:48.178 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:48.178 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:45:48.179 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:48.179 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:48.179 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_24 2022-04-10T01:45:48.179 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_24 deleted 2022-04-10T01:45:48.179 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:48.180 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:48.180 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_24 2022-04-10T01:45:48.180 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:45:48.180 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:48.181 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:48.181 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:48.181 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:48.181 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:48.181 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:48.182 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:48.182 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_24 2022-04-10T01:45:48.182 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:48.182 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:48.183 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_24 2022-04-10T01:45:48.183 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:48.183 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:48.183 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:48.183 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2022-04-10T01:45:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:48.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_24: (2) No such file or directory 2022-04-10T01:45:48.208 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:45:48.208 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:45:48.208 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:45:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_24 deleted '' 2022-04-10T01:45:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_24 2022-04-10T01:45:48.312 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:45:48.312 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:48.312 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:45:48.312 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:45:48.313 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:48.313 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:48.313 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:48.313 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:48.313 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_24 2022-04-10T01:45:48.314 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:48.314 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:48.314 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_24 2022-04-10T01:45:48.314 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2022-04-10T01:45:48.316 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:48.339 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_24: (2) No such file or directory 2022-04-10T01:45:48.344 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:45:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:45:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_25 2022-04-10T01:45:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_25 deleted 2022-04-10T01:45:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_25 2022-04-10T01:45:48.347 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:45:48.347 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:48.347 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:48.347 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_25 2022-04-10T01:45:48.349 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:48.349 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:48.349 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_25 2022-04-10T01:45:48.349 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2022-04-10T01:45:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:48.371 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_25: (2) No such file or directory 2022-04-10T01:45:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:45:48.374 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:45:48.374 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:45:48.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_25 deleted '' 2022-04-10T01:45:48.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:48.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:48.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_25 2022-04-10T01:45:48.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:45:48.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:48.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:45:48.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:45:48.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:48.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:48.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:48.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:48.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_25 2022-04-10T01:45:48.479 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_25 2022-04-10T01:45:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:48.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:48.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2022-04-10T01:45:48.481 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:48.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_25: (2) No such file or directory 2022-04-10T01:45:48.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:45:48.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:48.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:45:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_26 2022-04-10T01:45:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_26 deleted 2022-04-10T01:45:48.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:48.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:48.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_26 2022-04-10T01:45:48.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:45:48.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:48.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:48.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:48.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:48.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:48.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:48.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:48.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_26 2022-04-10T01:45:48.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:48.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_26 2022-04-10T01:45:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:48.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2022-04-10T01:45:48.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:48.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_26: (2) No such file or directory 2022-04-10T01:45:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:45:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:45:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:45:48.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_26 deleted '' 2022-04-10T01:45:48.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:48.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:48.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_26 2022-04-10T01:45:48.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:45:48.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:48.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:45:48.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:45:48.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:48.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:48.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:48.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_26 2022-04-10T01:45:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_26 2022-04-10T01:45:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2022-04-10T01:45:48.647 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:48.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_26: (2) No such file or directory 2022-04-10T01:45:48.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:45:48.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:48.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:45:48.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:48.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:48.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_27 2022-04-10T01:45:48.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_27 deleted 2022-04-10T01:45:48.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:48.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:48.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_27 2022-04-10T01:45:48.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:45:48.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:48.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:48.676 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:48.677 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:48.677 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:48.677 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:48.677 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:48.677 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_27 2022-04-10T01:45:48.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:48.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:48.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_27 2022-04-10T01:45:48.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:48.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:48.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:48.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:48.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2022-04-10T01:45:48.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:48.700 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_27: (2) No such file or directory 2022-04-10T01:45:48.702 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:45:48.703 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:45:48.703 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:45:48.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_27 deleted '' 2022-04-10T01:45:48.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:48.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:48.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_27 2022-04-10T01:45:48.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:45:48.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:48.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:45:48.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:45:48.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:48.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:48.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:48.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:48.809 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_27 2022-04-10T01:45:48.809 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:48.809 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:48.809 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_27 2022-04-10T01:45:48.810 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:48.810 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:48.810 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:48.810 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2022-04-10T01:45:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_27: (2) No such file or directory 2022-04-10T01:45:48.840 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:45:48.841 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:48.841 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:45:48.841 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:48.841 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:48.842 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_28 2022-04-10T01:45:48.842 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_28 deleted 2022-04-10T01:45:48.842 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:48.842 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:48.842 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_28 2022-04-10T01:45:48.843 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:45:48.843 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:48.843 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:48.843 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:48.844 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:48.844 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:48.844 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:48.844 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:48.845 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_28 2022-04-10T01:45:48.845 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:48.845 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:48.845 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_28 2022-04-10T01:45:48.845 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:48.846 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:48.846 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:48.846 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:48.846 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2022-04-10T01:45:48.847 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:48.869 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_28: (2) No such file or directory 2022-04-10T01:45:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:45:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:45:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:45:48.975 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_28 deleted '' 2022-04-10T01:45:48.975 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:48.975 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:48.976 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_28 2022-04-10T01:45:48.976 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:45:48.976 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:48.976 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:45:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:45:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:48.978 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:48.978 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:48.979 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:48.979 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_28 2022-04-10T01:45:48.979 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:48.979 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:48.980 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_28 2022-04-10T01:45:48.980 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:48.980 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:48.980 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:48.980 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2022-04-10T01:45:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_28: (2) No such file or directory 2022-04-10T01:45:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:45:49.008 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:49.008 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:45:49.008 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:49.009 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:49.009 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_29 2022-04-10T01:45:49.009 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_29 deleted 2022-04-10T01:45:49.009 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:49.009 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_29 2022-04-10T01:45:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:45:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:49.011 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:49.011 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:49.011 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:49.011 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:49.011 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:49.012 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_29 2022-04-10T01:45:49.012 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:49.012 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:49.012 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_29 2022-04-10T01:45:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2022-04-10T01:45:49.014 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:49.033 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_29: (2) No such file or directory 2022-04-10T01:45:49.036 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:45:49.036 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:45:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:45:49.139 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_29 deleted '' 2022-04-10T01:45:49.140 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:49.140 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:49.140 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_29 2022-04-10T01:45:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:45:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:45:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:45:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:49.142 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:49.142 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:49.142 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:49.142 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_29 2022-04-10T01:45:49.143 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:49.143 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:49.143 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_29 2022-04-10T01:45:49.143 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:49.143 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:49.144 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:49.144 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:49.144 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2022-04-10T01:45:49.144 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:49.165 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_29: (2) No such file or directory 2022-04-10T01:45:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:45:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:45:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_30 2022-04-10T01:45:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_30 deleted 2022-04-10T01:45:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_30 2022-04-10T01:45:49.173 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:45:49.173 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:49.173 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:49.173 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:49.173 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:49.174 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:49.174 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:49.174 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:49.174 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_30 2022-04-10T01:45:49.174 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:49.175 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:49.175 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_30 2022-04-10T01:45:49.175 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:49.175 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:49.175 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:49.176 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:49.176 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2022-04-10T01:45:49.176 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:49.198 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_30: (2) No such file or directory 2022-04-10T01:45:49.200 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:45:49.201 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:45:49.201 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:45:49.303 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_30 deleted '' 2022-04-10T01:45:49.303 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:49.304 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:49.304 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_30 2022-04-10T01:45:49.304 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:45:49.304 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:45:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:45:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:49.306 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:49.306 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:49.306 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_30 2022-04-10T01:45:49.306 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:49.306 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:49.307 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_30 2022-04-10T01:45:49.307 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:49.307 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:49.307 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:49.307 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:49.308 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2022-04-10T01:45:49.308 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_30: (2) No such file or directory 2022-04-10T01:45:49.336 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:45:49.336 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:49.337 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:45:49.337 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:49.337 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:49.337 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_31 2022-04-10T01:45:49.338 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_31 deleted 2022-04-10T01:45:49.338 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:49.338 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:49.338 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_31 2022-04-10T01:45:49.339 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:45:49.339 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:49.339 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:49.339 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:49.339 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:49.340 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:49.340 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:49.340 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:49.340 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_31 2022-04-10T01:45:49.341 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:49.341 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:49.341 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_31 2022-04-10T01:45:49.341 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:49.342 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:49.342 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:49.342 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:49.342 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2022-04-10T01:45:49.342 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:49.363 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_31: (2) No such file or directory 2022-04-10T01:45:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:45:49.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:45:49.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:45:49.472 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_31 deleted '' 2022-04-10T01:45:49.472 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:49.473 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:49.473 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_31 2022-04-10T01:45:49.473 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:45:49.473 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:45:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:45:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_31 2022-04-10T01:45:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_31 2022-04-10T01:45:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2022-04-10T01:45:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:49.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_31: (2) No such file or directory 2022-04-10T01:45:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:45:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:45:49.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:49.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:49.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_32 2022-04-10T01:45:49.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_32 deleted 2022-04-10T01:45:49.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:49.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:49.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_32 2022-04-10T01:45:49.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:45:49.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:49.505 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:49.505 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:49.505 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:49.505 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_32 2022-04-10T01:45:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_32 2022-04-10T01:45:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2022-04-10T01:45:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:49.528 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_32: (2) No such file or directory 2022-04-10T01:45:49.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:45:49.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:45:49.532 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:45:49.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_32 deleted '' 2022-04-10T01:45:49.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:49.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_32 2022-04-10T01:45:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:45:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:45:49.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:45:49.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:49.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:49.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:49.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:49.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_32 2022-04-10T01:45:49.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:49.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:49.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_32 2022-04-10T01:45:49.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:49.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:49.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:49.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:49.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2022-04-10T01:45:49.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_32: (2) No such file or directory 2022-04-10T01:45:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:45:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:45:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_33 2022-04-10T01:45:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_33 deleted 2022-04-10T01:45:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_33 2022-04-10T01:45:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:45:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_33 2022-04-10T01:45:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_33 2022-04-10T01:45:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2022-04-10T01:45:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:49.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_33: (2) No such file or directory 2022-04-10T01:45:49.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:45:49.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:45:49.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:45:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_33 deleted '' 2022-04-10T01:45:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:49.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_33 2022-04-10T01:45:49.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:45:49.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:49.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:45:49.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:45:49.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:49.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:49.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:49.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_33 2022-04-10T01:45:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_33 2022-04-10T01:45:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2022-04-10T01:45:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:49.831 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_33: (2) No such file or directory 2022-04-10T01:45:49.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:45:49.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:49.837 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:45:49.837 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:49.837 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:49.837 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_34 2022-04-10T01:45:49.838 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_34 deleted 2022-04-10T01:45:49.838 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:49.838 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:49.838 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_34 2022-04-10T01:45:49.839 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:45:49.839 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:49.839 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:49.839 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:49.839 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:49.840 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:49.840 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:49.840 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:49.840 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_34 2022-04-10T01:45:49.840 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:49.841 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:49.841 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_34 2022-04-10T01:45:49.841 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:49.841 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2022-04-10T01:45:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:49.865 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_34: (2) No such file or directory 2022-04-10T01:45:49.868 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:45:49.868 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:45:49.868 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:45:49.971 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_34 deleted '' 2022-04-10T01:45:49.971 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:49.971 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:49.972 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_34 2022-04-10T01:45:49.972 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:45:49.972 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:49.972 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:45:49.972 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:45:49.973 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:49.973 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:49.973 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:49.973 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_34 2022-04-10T01:45:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_34 2022-04-10T01:45:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:49.975 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:49.975 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:49.975 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:49.975 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2022-04-10T01:45:49.976 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:49.999 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_34: (2) No such file or directory 2022-04-10T01:45:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:45:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:45:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_35 2022-04-10T01:45:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_35 deleted 2022-04-10T01:45:50.006 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:50.006 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:50.006 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_35 2022-04-10T01:45:50.006 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:45:50.007 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:50.007 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:50.007 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:50.007 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:50.007 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_35 2022-04-10T01:45:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:50.009 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:50.009 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_35 2022-04-10T01:45:50.009 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:50.009 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:50.009 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2022-04-10T01:45:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:50.031 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_35: (2) No such file or directory 2022-04-10T01:45:50.034 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:45:50.034 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:45:50.034 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:45:50.136 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_35 deleted '' 2022-04-10T01:45:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_35 2022-04-10T01:45:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:45:50.138 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:50.138 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:45:50.138 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:45:50.138 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:50.139 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:50.139 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:50.139 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:50.139 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_35 2022-04-10T01:45:50.139 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:50.140 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:50.140 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_35 2022-04-10T01:45:50.140 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:50.140 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:50.141 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:50.141 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:50.141 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2022-04-10T01:45:50.141 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:50.165 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_35: (2) No such file or directory 2022-04-10T01:45:50.170 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:45:50.170 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:50.170 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:45:50.171 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:50.171 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:50.171 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_36 2022-04-10T01:45:50.171 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_36 deleted 2022-04-10T01:45:50.171 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:50.172 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:50.172 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_36 2022-04-10T01:45:50.172 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:45:50.172 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:50.172 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:50.173 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:50.173 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:50.173 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:50.173 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:50.173 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_36 2022-04-10T01:45:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_36 2022-04-10T01:45:50.175 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:50.175 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:50.175 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:50.175 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:50.175 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2022-04-10T01:45:50.176 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:50.197 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_36: (2) No such file or directory 2022-04-10T01:45:50.200 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:45:50.201 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:45:50.201 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:45:50.304 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_36 deleted '' 2022-04-10T01:45:50.305 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:50.305 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:50.305 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_36 2022-04-10T01:45:50.306 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:45:50.306 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:50.307 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:45:50.307 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:45:50.307 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:50.307 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_36 2022-04-10T01:45:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_36 2022-04-10T01:45:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:50.310 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:50.310 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:50.310 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:50.310 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2022-04-10T01:45:50.310 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:50.334 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_36: (2) No such file or directory 2022-04-10T01:45:50.339 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:45:50.339 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:50.339 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:45:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_37 2022-04-10T01:45:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_37 deleted 2022-04-10T01:45:50.341 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:50.341 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:50.341 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_37 2022-04-10T01:45:50.341 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:45:50.341 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:50.342 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:50.342 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:50.342 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:50.342 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:50.343 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:50.343 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:50.343 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_37 2022-04-10T01:45:50.343 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:50.343 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:50.344 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_37 2022-04-10T01:45:50.344 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:50.344 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:50.344 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:50.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:50.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2022-04-10T01:45:50.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:50.366 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_37: (2) No such file or directory 2022-04-10T01:45:50.369 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:45:50.369 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:45:50.369 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:45:50.472 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_37 deleted '' 2022-04-10T01:45:50.472 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:50.472 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:50.472 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_37 2022-04-10T01:45:50.473 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:45:50.473 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:50.473 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:45:50.473 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:45:50.474 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:50.474 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:50.474 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:50.474 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:50.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_37 2022-04-10T01:45:50.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:50.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:50.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_37 2022-04-10T01:45:50.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:50.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:50.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:50.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:50.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2022-04-10T01:45:50.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:50.499 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_37: (2) No such file or directory 2022-04-10T01:45:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:45:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:45:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:50.505 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_38 2022-04-10T01:45:50.505 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_38 deleted 2022-04-10T01:45:50.505 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:50.505 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:50.506 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_38 2022-04-10T01:45:50.506 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:45:50.506 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:50.506 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:50.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:50.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:50.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:50.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:50.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:50.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_38 2022-04-10T01:45:50.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:50.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:50.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_38 2022-04-10T01:45:50.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:50.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:50.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:50.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:50.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2022-04-10T01:45:50.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:50.530 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_38: (2) No such file or directory 2022-04-10T01:45:50.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:45:50.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:45:50.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:45:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_38 deleted '' 2022-04-10T01:45:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_38 2022-04-10T01:45:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:45:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:45:50.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:45:50.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:50.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:50.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:50.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_38 2022-04-10T01:45:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_38 2022-04-10T01:45:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:50.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:50.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:50.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:50.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2022-04-10T01:45:50.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_38: (2) No such file or directory 2022-04-10T01:45:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:45:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:45:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:50.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:50.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_39 2022-04-10T01:45:50.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_39 deleted 2022-04-10T01:45:50.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:50.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:50.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_39 2022-04-10T01:45:50.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:45:50.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:50.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:50.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:50.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:50.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:50.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:50.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:50.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_39 2022-04-10T01:45:50.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:50.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:50.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_39 2022-04-10T01:45:50.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:50.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:50.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:50.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:50.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2022-04-10T01:45:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:50.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_39: (2) No such file or directory 2022-04-10T01:45:50.698 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:45:50.699 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:45:50.699 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:45:50.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_39 deleted '' 2022-04-10T01:45:50.801 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:50.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:50.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_39 2022-04-10T01:45:50.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:45:50.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:50.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:45:50.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:45:50.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:50.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:50.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:50.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:50.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_39 2022-04-10T01:45:50.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:50.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:50.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_39 2022-04-10T01:45:50.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:50.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:50.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:50.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:50.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2022-04-10T01:45:50.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:50.832 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_39: (2) No such file or directory 2022-04-10T01:45:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:45:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:45:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_40 2022-04-10T01:45:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_40 deleted 2022-04-10T01:45:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_40 2022-04-10T01:45:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:45:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_40 2022-04-10T01:45:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_40 2022-04-10T01:45:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2022-04-10T01:45:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:50.871 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_40: (2) No such file or directory 2022-04-10T01:45:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:45:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:45:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:45:50.976 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_40 deleted '' 2022-04-10T01:45:50.977 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:50.977 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:50.977 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_40 2022-04-10T01:45:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:45:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:45:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:45:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_40 2022-04-10T01:45:50.980 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:50.980 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:50.980 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_40 2022-04-10T01:45:50.980 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:50.981 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:50.981 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:50.981 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:50.981 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2022-04-10T01:45:50.981 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:51.003 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_40: (2) No such file or directory 2022-04-10T01:45:51.008 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:45:51.009 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:51.009 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:45:51.009 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:51.009 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:51.010 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_41 2022-04-10T01:45:51.010 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_41 deleted 2022-04-10T01:45:51.010 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:51.010 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:51.010 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_41 2022-04-10T01:45:51.011 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:45:51.011 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:51.011 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:51.011 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:51.011 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:51.012 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:51.012 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:51.012 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:51.012 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_41 2022-04-10T01:45:51.012 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_41 2022-04-10T01:45:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:51.014 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:51.014 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2022-04-10T01:45:51.014 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_41: (2) No such file or directory 2022-04-10T01:45:51.038 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:45:51.039 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:45:51.039 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:45:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_41 deleted '' 2022-04-10T01:45:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_41 2022-04-10T01:45:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:45:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:45:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:45:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_41 2022-04-10T01:45:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_41 2022-04-10T01:45:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2022-04-10T01:45:51.146 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:51.169 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_41: (2) No such file or directory 2022-04-10T01:45:51.174 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:45:51.174 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:51.174 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:45:51.175 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:51.175 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:51.175 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_42 2022-04-10T01:45:51.175 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_42 deleted 2022-04-10T01:45:51.176 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:51.176 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:51.176 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_42 2022-04-10T01:45:51.176 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:45:51.176 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:51.177 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:51.177 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:51.177 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:51.177 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:51.177 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:51.178 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:51.178 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_42 2022-04-10T01:45:51.178 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:51.178 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:51.179 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_42 2022-04-10T01:45:51.179 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:51.179 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:51.179 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:51.179 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:51.180 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2022-04-10T01:45:51.180 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:51.201 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_42: (2) No such file or directory 2022-04-10T01:45:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:45:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:45:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:45:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_42 deleted '' 2022-04-10T01:45:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:51.308 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_42 2022-04-10T01:45:51.308 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:45:51.308 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:51.308 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:45:51.308 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:45:51.309 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:51.309 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:51.309 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:51.309 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:51.309 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_42 2022-04-10T01:45:51.310 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:51.310 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:51.310 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_42 2022-04-10T01:45:51.310 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2022-04-10T01:45:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:51.335 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_42: (2) No such file or directory 2022-04-10T01:45:51.340 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:45:51.340 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:51.340 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:45:51.340 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_43 2022-04-10T01:45:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_43 deleted 2022-04-10T01:45:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_43 2022-04-10T01:45:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:45:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:51.343 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:51.343 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:51.343 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:51.343 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_43 2022-04-10T01:45:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_43 2022-04-10T01:45:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2022-04-10T01:45:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:51.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_43: (2) No such file or directory 2022-04-10T01:45:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:45:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:45:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:45:51.473 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_43 deleted '' 2022-04-10T01:45:51.473 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:51.474 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:51.474 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_43 2022-04-10T01:45:51.474 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:45:51.474 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:51.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:45:51.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:45:51.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:51.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:51.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:51.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:51.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_43 2022-04-10T01:45:51.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:51.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:51.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_43 2022-04-10T01:45:51.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:51.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:51.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:51.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:51.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2022-04-10T01:45:51.478 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:51.501 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_43: (2) No such file or directory 2022-04-10T01:45:51.506 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:45:51.506 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:51.506 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:45:51.506 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:51.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:51.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_44 2022-04-10T01:45:51.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_44 deleted 2022-04-10T01:45:51.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:51.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:51.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_44 2022-04-10T01:45:51.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:45:51.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:51.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:51.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:51.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:51.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:51.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:51.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:51.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_44 2022-04-10T01:45:51.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:51.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:51.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_44 2022-04-10T01:45:51.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:51.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:51.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:51.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:51.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2022-04-10T01:45:51.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:51.532 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_44: (2) No such file or directory 2022-04-10T01:45:51.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:45:51.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:45:51.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:45:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_44 deleted '' 2022-04-10T01:45:51.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:51.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:51.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_44 2022-04-10T01:45:51.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:45:51.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:51.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:45:51.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:45:51.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:51.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:51.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:51.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:51.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_44 2022-04-10T01:45:51.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:51.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:51.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_44 2022-04-10T01:45:51.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:51.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:51.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2022-04-10T01:45:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_44: (2) No such file or directory 2022-04-10T01:45:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:45:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:51.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:45:51.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:51.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:51.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_45 2022-04-10T01:45:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_45 deleted 2022-04-10T01:45:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_45 2022-04-10T01:45:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:45:51.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:51.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:51.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:51.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:51.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:51.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:51.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:51.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_45 2022-04-10T01:45:51.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:51.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:51.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_45 2022-04-10T01:45:51.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:51.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:51.674 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:51.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:51.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2022-04-10T01:45:51.675 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:51.696 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_45: (2) No such file or directory 2022-04-10T01:45:51.699 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:45:51.699 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:45:51.700 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:45:51.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_45 deleted '' 2022-04-10T01:45:51.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:51.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:51.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_45 2022-04-10T01:45:51.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:45:51.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:51.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:45:51.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:45:51.804 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:51.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:51.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:51.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:51.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_45 2022-04-10T01:45:51.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:51.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:51.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_45 2022-04-10T01:45:51.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:51.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:51.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:51.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:51.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2022-04-10T01:45:51.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_45: (2) No such file or directory 2022-04-10T01:45:51.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:45:51.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:51.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:45:51.836 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:51.837 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:51.837 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_46 2022-04-10T01:45:51.837 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_46 deleted 2022-04-10T01:45:51.837 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:51.837 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:51.838 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_46 2022-04-10T01:45:51.838 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:45:51.838 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:51.838 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:51.839 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:51.839 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:51.839 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:51.839 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:51.840 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:51.840 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_46 2022-04-10T01:45:51.840 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:51.840 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:51.840 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_46 2022-04-10T01:45:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:51.842 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2022-04-10T01:45:51.842 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:51.867 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_46: (2) No such file or directory 2022-04-10T01:45:51.870 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:45:51.870 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:45:51.871 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:45:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_46 deleted '' 2022-04-10T01:45:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_46 2022-04-10T01:45:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:45:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:45:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:45:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_46 2022-04-10T01:45:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_46 2022-04-10T01:45:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2022-04-10T01:45:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_46: (2) No such file or directory 2022-04-10T01:45:52.006 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:45:52.006 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:52.007 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:45:52.007 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:52.007 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:52.007 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_47 2022-04-10T01:45:52.008 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_47 deleted 2022-04-10T01:45:52.008 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:52.008 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:52.008 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_47 2022-04-10T01:45:52.009 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:45:52.009 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:52.009 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:52.009 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:52.009 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:52.010 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:52.010 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:52.010 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:52.010 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_47 2022-04-10T01:45:52.011 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:52.011 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:52.011 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_47 2022-04-10T01:45:52.011 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:52.011 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:52.012 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:52.012 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:52.012 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2022-04-10T01:45:52.012 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:52.033 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_47: (2) No such file or directory 2022-04-10T01:45:52.035 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:45:52.036 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:45:52.036 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:45:52.138 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_47 deleted '' 2022-04-10T01:45:52.138 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:52.139 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:52.139 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_47 2022-04-10T01:45:52.139 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:45:52.139 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:52.140 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:45:52.140 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:45:52.140 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:52.140 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:52.140 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:52.141 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:52.141 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_47 2022-04-10T01:45:52.141 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:52.141 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:52.141 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_47 2022-04-10T01:45:52.142 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:52.142 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:52.142 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:52.142 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:52.143 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2022-04-10T01:45:52.143 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:52.165 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_47: (2) No such file or directory 2022-04-10T01:45:52.170 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:45:52.170 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:52.170 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:45:52.170 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:52.171 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:52.171 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_48 2022-04-10T01:45:52.171 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_48 deleted 2022-04-10T01:45:52.171 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:52.172 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:52.172 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_48 2022-04-10T01:45:52.172 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:45:52.172 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:52.173 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:52.173 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:52.173 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:52.173 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:52.173 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_48 2022-04-10T01:45:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_48 2022-04-10T01:45:52.175 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:52.175 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:52.175 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:52.175 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:52.175 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2022-04-10T01:45:52.176 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:52.197 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_48: (2) No such file or directory 2022-04-10T01:45:52.200 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:45:52.200 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:45:52.200 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:45:52.302 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_48 deleted '' 2022-04-10T01:45:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_48 2022-04-10T01:45:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:45:52.304 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:52.304 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:45:52.304 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:45:52.304 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:52.305 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:52.305 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:52.305 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:52.305 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_48 2022-04-10T01:45:52.305 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:52.306 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:52.306 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_48 2022-04-10T01:45:52.306 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:52.306 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:52.306 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:52.307 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:52.307 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2022-04-10T01:45:52.307 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:52.331 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_48: (2) No such file or directory 2022-04-10T01:45:52.335 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:45:52.336 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:52.336 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:45:52.336 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:52.336 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:52.337 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_49 2022-04-10T01:45:52.337 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_49 deleted 2022-04-10T01:45:52.337 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:52.337 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:52.337 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_49 2022-04-10T01:45:52.338 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:45:52.338 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:52.338 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:52.338 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:52.339 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:52.339 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:52.339 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:52.339 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:52.339 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_49 2022-04-10T01:45:52.340 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:52.340 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:52.340 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_49 2022-04-10T01:45:52.340 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:52.340 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:52.341 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:52.341 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:52.341 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2022-04-10T01:45:52.341 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:52.364 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_49: (2) No such file or directory 2022-04-10T01:45:52.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:45:52.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:45:52.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:45:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_49 deleted '' 2022-04-10T01:45:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_49 2022-04-10T01:45:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:45:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:45:52.472 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:45:52.472 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:52.472 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:52.472 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:52.473 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:52.473 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_49 2022-04-10T01:45:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_49 2022-04-10T01:45:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2022-04-10T01:45:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_49: (2) No such file or directory 2022-04-10T01:45:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:45:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:45:52.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:52.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2022-04-10T01:45:52.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_50 2022-04-10T01:45:52.503 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_50 deleted 2022-04-10T01:45:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_50 2022-04-10T01:45:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:45:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:45:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_50 2022-04-10T01:45:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_50 2022-04-10T01:45:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:52.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:52.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2022-04-10T01:45:52.508 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:52.529 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_50: (2) No such file or directory 2022-04-10T01:45:52.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:45:52.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:45:52.532 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:45:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_50 deleted '' 2022-04-10T01:45:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:52.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_50 2022-04-10T01:45:52.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:45:52.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:45:52.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:45:52.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:45:52.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:52.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:52.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:52.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:52.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_50 2022-04-10T01:45:52.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:52.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:45:52.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_50 2022-04-10T01:45:52.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:45:52.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:45:52.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:52.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:45:52.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2022-04-10T01:45:52.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:45:52.661 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_50: (2) No such file or directory 2022-04-10T01:45:52.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:45:52.664 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:45:52.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:45:52.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:52.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ testlog 'TEST: delete images during bootstrap' 2022-04-10T01:45:52.665 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ tee -a /tmp/tmp.rgwz74WspX/rbd-mirror.test.log 2022-04-10T01:45:52.666 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ date '+%F %T' 2022-04-10T01:45:52.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo 2022-04-10 01:45:52 TEST: delete images during bootstrap 2022-04-10T01:45:52.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:2022-04-10 01:45:52 TEST: delete images during bootstrap 2022-04-10T01:45:52.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set_pool_mirror_mode cluster1 mirror image 2022-04-10T01:45:52.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:52.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:52.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=image 2022-04-10T01:45:52.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:52.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:52.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:52.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:52.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster1 mirror pool enable mirror image 2022-04-10T01:45:52.898 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd: mirroring is already configured for image mode 2022-04-10T01:45:52.901 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set_pool_mirror_mode cluster2 mirror image 2022-04-10T01:45:52.901 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:52.901 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:52.901 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=image 2022-04-10T01:45:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror image 2022-04-10T01:45:52.928 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd: mirroring is already configured for image mode 2022-04-10T01:45:52.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ start_mirror cluster1 2022-04-10T01:45:52.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:52.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local instance 2022-04-10T01:45:52.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set_cluster_instance cluster1 cluster instance 2022-04-10T01:45:52.935 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local val=cluster1 2022-04-10T01:45:52.935 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster_var_name=cluster 2022-04-10T01:45:52.935 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local instance_var_name=instance 2022-04-10T01:45:52.935 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cluster=cluster1 2022-04-10T01:45:52.936 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ instance=cluster1 2022-04-10T01:45:52.936 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster1 ']' 2022-04-10T01:45:52.936 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ instance=0 2022-04-10T01:45:52.936 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ eval cluster=cluster1 2022-04-10T01:45:52.937 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cluster=cluster1 2022-04-10T01:45:52.937 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ eval instance=0 2022-04-10T01:45:52.937 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ instance=0 2022-04-10T01:45:52.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:45:52.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:45:52.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n 1 2022-04-10T01:45:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 2022-04-10T01:45:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=test 2022-04-10T01:45:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ seq 1 10 2022-04-10T01:45:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 10` 2022-04-10T01:45:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_1 2022-04-10T01:45:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_1 512M 2022-04-10T01:45:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:52.941 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:45:52.941 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:52.941 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:45:52.941 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_1 2022-04-10T01:45:52.941 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:45:52.942 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:45:52.942 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:52.942 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:52.942 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:52.942 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:52.943 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 512M ']' 2022-04-10T01:45:52.943 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=512M 2022-04-10T01:45:52.943 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:45:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_1 2022-04-10T01:45:52.976 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_1 2022-04-10T01:45:52.976 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_1 2022-04-10T01:45:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:45:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2022-04-10T01:45:53.201 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.34773902003091695) > weight (0.25) 2022-04-10T01:45:53.202 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.6317499261968654) > weight (0.25) 2022-04-10T01:45:53.202 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.3425820029885315) > weight (0.25) 2022-04-10T01:45:53.202 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.3215775311199478) > weight (0.25) 2022-04-10T01:45:53.203 INFO:tasks.rbd_mirror_thrash:waiting for 73 secs before thrashing 2022-04-10T01:45:53.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:45:53.587 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2022-04-10T01:45:53.621 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_1': 2022-04-10T01:45:53.621 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 512 MiB in 128 objects 2022-04-10T01:45:53.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:45:53.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:45:53.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 18662ecd80cb 2022-04-10T01:45:53.622 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.18662ecd80cb 2022-04-10T01:45:53.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:45:53.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:45:53.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:45:53.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:45:53.625 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:45:52 2022 2022-04-10T01:45:53.625 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:45:52 2022 2022-04-10T01:45:53.625 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:45:52 2022 2022-04-10T01:45:53.625 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:45:53.626 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:45:53.626 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 53c54804-c4f0-4217-a937-1223c63bfc97 2022-04-10T01:45:53.626 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:45:53.628 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_1 2022-04-10T01:45:53.628 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:53.628 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:45:53.628 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_1 2022-04-10T01:45:53.628 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:45:53.629 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=9 2022-04-10T01:45:53.629 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:45:53.629 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:45:53.630 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:45:53.630 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:45:53.630 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:45:53.630 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:45:53.812 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 35544064 bytes 2022-04-10T01:45:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 39387136 bytes 2022-04-10T01:45:54.089 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 104570880 bytes 2022-04-10T01:45:54.275 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 170397696 bytes 2022-04-10T01:45:54.409 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 207868928 bytes 2022-04-10T01:45:54.619 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 277058560 bytes 2022-04-10T01:45:54.744 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 293978112 bytes 2022-04-10T01:45:54.877 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 320602112 bytes 2022-04-10T01:45:55.099 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 421559296 bytes 2022-04-10T01:45:55.223 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 441021440 bytes 2022-04-10T01:45:55.331 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 453329920 bytes 2022-04-10T01:45:55.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 556550144 bytes 2022-04-10T01:45:55.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 570016768 bytes 2022-04-10T01:45:55.791 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 571108352 bytes 2022-04-10T01:45:55.973 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 630429696 bytes 2022-04-10T01:45:56.157 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 721397760 bytes 2022-04-10T01:45:56.299 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 722259968 bytes 2022-04-10T01:45:56.422 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 743652352 bytes 2022-04-10T01:45:56.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 749238272 bytes 2022-04-10T01:45:56.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 795591680 bytes 2022-04-10T01:45:56.856 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 840715264 bytes 2022-04-10T01:45:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 910143488 bytes 2022-04-10T01:45:57.234 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 978957312 bytes 2022-04-10T01:45:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 979591168 bytes 2022-04-10T01:45:57.463 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 992062464 bytes 2022-04-10T01:45:57.571 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 992667648 bytes 2022-04-10T01:45:57.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1025852416 bytes 2022-04-10T01:45:57.798 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr:2022-04-10T01:45:57.793+0000 7f53052f5700 -1 bdev(0x5600de1c6000 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2022-04-10T01:45:57.798 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11492-g9b3739e7/rpm/el8/BUILD/ceph-17.0.0-11492-g9b3739e7/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f53052f5700 time 2022-04-10T01:45:57.794795+0000 2022-04-10T01:45:57.799 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11492-g9b3739e7/rpm/el8/BUILD/ceph-17.0.0-11492-g9b3739e7/src/blk/kernel/KernelDevice.cc: 621: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2022-04-10T01:45:57.799 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: ceph version 17.0.0-11492-g9b3739e7 (9b3739e792ef182b79445b926c7b7da461c05f52) quincy (dev) 2022-04-10T01:45:57.799 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x5600db2490cb] 2022-04-10T01:45:57.800 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 2: (KernelDevice::_aio_thread()+0xf9e) [0x5600dbe9b81e] 2022-04-10T01:45:57.800 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5600dbea9051] 2022-04-10T01:45:57.800 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 4: /lib64/libpthread.so.0(+0x81cf) [0x7f5314f481cf] 2022-04-10T01:45:57.800 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 5: clone() 2022-04-10T01:45:57.800 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr:*** Caught signal (Aborted) ** 2022-04-10T01:45:57.801 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: in thread 7f53052f5700 thread_name:bstore_aio 2022-04-10T01:45:57.801 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr:2022-04-10T01:45:57.795+0000 7f53052f5700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11492-g9b3739e7/rpm/el8/BUILD/ceph-17.0.0-11492-g9b3739e7/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f53052f5700 time 2022-04-10T01:45:57.794795+0000 2022-04-10T01:45:57.801 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11492-g9b3739e7/rpm/el8/BUILD/ceph-17.0.0-11492-g9b3739e7/src/blk/kernel/KernelDevice.cc: 621: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2022-04-10T01:45:57.801 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 2022-04-10T01:45:57.802 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: ceph version 17.0.0-11492-g9b3739e7 (9b3739e792ef182b79445b926c7b7da461c05f52) quincy (dev) 2022-04-10T01:45:57.802 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x5600db2490cb] 2022-04-10T01:45:57.802 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 2: (KernelDevice::_aio_thread()+0xf9e) [0x5600dbe9b81e] 2022-04-10T01:45:57.802 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5600dbea9051] 2022-04-10T01:45:57.803 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 4: /lib64/libpthread.so.0(+0x81cf) [0x7f5314f481cf] 2022-04-10T01:45:57.803 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 5: clone() 2022-04-10T01:45:57.803 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 2022-04-10T01:45:57.804 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: ceph version 17.0.0-11492-g9b3739e7 (9b3739e792ef182b79445b926c7b7da461c05f52) quincy (dev) 2022-04-10T01:45:57.804 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f5314f52ce0] 2022-04-10T01:45:57.804 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 2: gsignal() 2022-04-10T01:45:57.805 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 3: abort() 2022-04-10T01:45:57.805 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b8) [0x5600db24919e] 2022-04-10T01:45:57.805 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 5: (KernelDevice::_aio_thread()+0xf9e) [0x5600dbe9b81e] 2022-04-10T01:45:57.805 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5600dbea9051] 2022-04-10T01:45:57.806 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 7: /lib64/libpthread.so.0(+0x81cf) [0x7f5314f481cf] 2022-04-10T01:45:57.806 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 8: clone() 2022-04-10T01:45:57.806 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr:2022-04-10T01:45:57.797+0000 7f53052f5700 -1 *** Caught signal (Aborted) ** 2022-04-10T01:45:57.807 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: in thread 7f53052f5700 thread_name:bstore_aio 2022-04-10T01:45:57.807 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 2022-04-10T01:45:57.807 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: ceph version 17.0.0-11492-g9b3739e7 (9b3739e792ef182b79445b926c7b7da461c05f52) quincy (dev) 2022-04-10T01:45:57.807 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f5314f52ce0] 2022-04-10T01:45:57.807 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 2: gsignal() 2022-04-10T01:45:57.808 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 3: abort() 2022-04-10T01:45:57.808 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b8) [0x5600db24919e] 2022-04-10T01:45:57.808 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 5: (KernelDevice::_aio_thread()+0xf9e) [0x5600dbe9b81e] 2022-04-10T01:45:57.808 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5600dbea9051] 2022-04-10T01:45:57.809 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 7: /lib64/libpthread.so.0(+0x81cf) [0x7f5314f481cf] 2022-04-10T01:45:57.809 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 8: clone() 2022-04-10T01:45:57.809 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-04-10T01:45:57.809 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 2022-04-10T01:45:57.817 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: -1179> 2022-04-10T01:45:57.793+0000 7f53052f5700 -1 bdev(0x5600de1c6000 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2022-04-10T01:45:57.817 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: -685> 2022-04-10T01:45:57.795+0000 7f53052f5700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11492-g9b3739e7/rpm/el8/BUILD/ceph-17.0.0-11492-g9b3739e7/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f53052f5700 time 2022-04-10T01:45:57.794795+0000 2022-04-10T01:45:57.818 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11492-g9b3739e7/rpm/el8/BUILD/ceph-17.0.0-11492-g9b3739e7/src/blk/kernel/KernelDevice.cc: 621: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2022-04-10T01:45:57.818 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 2022-04-10T01:45:57.818 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: ceph version 17.0.0-11492-g9b3739e7 (9b3739e792ef182b79445b926c7b7da461c05f52) quincy (dev) 2022-04-10T01:45:57.819 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x5600db2490cb] 2022-04-10T01:45:57.819 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 2: (KernelDevice::_aio_thread()+0xf9e) [0x5600dbe9b81e] 2022-04-10T01:45:57.819 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5600dbea9051] 2022-04-10T01:45:57.819 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 4: /lib64/libpthread.so.0(+0x81cf) [0x7f5314f481cf] 2022-04-10T01:45:57.820 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 5: clone() 2022-04-10T01:45:57.820 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 2022-04-10T01:45:57.821 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 0> 2022-04-10T01:45:57.797+0000 7f53052f5700 -1 *** Caught signal (Aborted) ** 2022-04-10T01:45:57.821 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: in thread 7f53052f5700 thread_name:bstore_aio 2022-04-10T01:45:57.821 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 2022-04-10T01:45:57.821 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: ceph version 17.0.0-11492-g9b3739e7 (9b3739e792ef182b79445b926c7b7da461c05f52) quincy (dev) 2022-04-10T01:45:57.822 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f5314f52ce0] 2022-04-10T01:45:57.822 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 2: gsignal() 2022-04-10T01:45:57.822 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 3: abort() 2022-04-10T01:45:57.823 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b8) [0x5600db24919e] 2022-04-10T01:45:57.823 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 5: (KernelDevice::_aio_thread()+0xf9e) [0x5600dbe9b81e] 2022-04-10T01:45:57.823 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5600dbea9051] 2022-04-10T01:45:57.823 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 7: /lib64/libpthread.so.0(+0x81cf) [0x7f5314f481cf] 2022-04-10T01:45:57.824 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 8: clone() 2022-04-10T01:45:57.824 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-04-10T01:45:57.824 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 2022-04-10T01:45:57.827 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: -2180> 2022-04-10T01:45:57.793+0000 7f53052f5700 -1 bdev(0x5600de1c6000 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2022-04-10T01:45:57.828 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: -1686> 2022-04-10T01:45:57.795+0000 7f53052f5700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11492-g9b3739e7/rpm/el8/BUILD/ceph-17.0.0-11492-g9b3739e7/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f53052f5700 time 2022-04-10T01:45:57.794795+0000 2022-04-10T01:45:57.828 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11492-g9b3739e7/rpm/el8/BUILD/ceph-17.0.0-11492-g9b3739e7/src/blk/kernel/KernelDevice.cc: 621: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2022-04-10T01:45:57.828 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 2022-04-10T01:45:57.829 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: ceph version 17.0.0-11492-g9b3739e7 (9b3739e792ef182b79445b926c7b7da461c05f52) quincy (dev) 2022-04-10T01:45:57.829 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x5600db2490cb] 2022-04-10T01:45:57.829 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 2: (KernelDevice::_aio_thread()+0xf9e) [0x5600dbe9b81e] 2022-04-10T01:45:57.829 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5600dbea9051] 2022-04-10T01:45:57.830 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 4: /lib64/libpthread.so.0(+0x81cf) [0x7f5314f481cf] 2022-04-10T01:45:57.830 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 5: clone() 2022-04-10T01:45:57.830 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 2022-04-10T01:45:57.831 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: -1001> 2022-04-10T01:45:57.797+0000 7f53052f5700 -1 *** Caught signal (Aborted) ** 2022-04-10T01:45:57.831 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: in thread 7f53052f5700 thread_name:bstore_aio 2022-04-10T01:45:57.831 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 2022-04-10T01:45:57.831 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: ceph version 17.0.0-11492-g9b3739e7 (9b3739e792ef182b79445b926c7b7da461c05f52) quincy (dev) 2022-04-10T01:45:57.832 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f5314f52ce0] 2022-04-10T01:45:57.832 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 2: gsignal() 2022-04-10T01:45:57.832 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 3: abort() 2022-04-10T01:45:57.832 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b8) [0x5600db24919e] 2022-04-10T01:45:57.833 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 5: (KernelDevice::_aio_thread()+0xf9e) [0x5600dbe9b81e] 2022-04-10T01:45:57.833 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5600dbea9051] 2022-04-10T01:45:57.833 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 7: /lib64/libpthread.so.0(+0x81cf) [0x7f5314f481cf] 2022-04-10T01:45:57.833 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 8: clone() 2022-04-10T01:45:57.834 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-04-10T01:45:57.834 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr: 2022-04-10T01:45:58.005 INFO:tasks.ceph.cluster2.osd.0.smithi043.stderr:daemon-helper: command failed with exit status 5 2022-04-10T01:45:59.700 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1072345088 bytes 2022-04-10T01:45:59.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1106341888 bytes 2022-04-10T01:45:59.895 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1140812800 bytes 2022-04-10T01:46:00.013 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1184470016 bytes 2022-04-10T01:46:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1252064256 bytes 2022-04-10T01:46:00.264 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1322444800 bytes 2022-04-10T01:46:00.400 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1379118080 bytes 2022-04-10T01:46:00.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1419561984 bytes 2022-04-10T01:46:00.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1443595264 bytes 2022-04-10T01:46:00.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1492273152 bytes 2022-04-10T01:46:00.844 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1576107008 bytes 2022-04-10T01:46:00.929 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~0s 2022-04-10T01:46:00.937 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1596899328 bytes 2022-04-10T01:46:01.071 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1695944704 bytes 2022-04-10T01:46:01.185 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1730010112 bytes 2022-04-10T01:46:01.289 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 1759258624 bytes 2022-04-10T01:46:01.375 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 1776233472 bytes 2022-04-10T01:46:01.483 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 1858689024 bytes 2022-04-10T01:46:01.614 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 1882775552 bytes 2022-04-10T01:46:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 1913418752 bytes 2022-04-10T01:46:01.825 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 1954074624 bytes 2022-04-10T01:46:01.913 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 1971842048 bytes 2022-04-10T01:46:01.981 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 1972448256 bytes 2022-04-10T01:46:02.105 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 2030498816 bytes 2022-04-10T01:46:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 2105711616 bytes 2022-04-10T01:46:02.321 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 2117750784 bytes 2022-04-10T01:46:02.423 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 2158277632 bytes 2022-04-10T01:46:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5400 IOs, 2167246848 bytes 2022-04-10T01:46:02.588 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5500 IOs, 2192289792 bytes 2022-04-10T01:46:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:46:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:46:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:46:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:46:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_1 present 2022-04-10T01:46:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:46:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:46:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_1 2022-04-10T01:46:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=present 2022-04-10T01:46:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:46:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:46:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:46:02.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:46:02.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:46:02.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:46:02.636 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:46:02.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_1 2022-04-10T01:46:02.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:46:02.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:46:02.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_1 2022-04-10T01:46:02.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:46:02.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:46:02.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:46:02.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:46:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2022-04-10T01:46:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:46:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id=113399b8ea92 2022-04-10T01:46:02.876 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:46:02.876 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:46:02.978 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_1 present 113399b8ea92 2022-04-10T01:46:02.979 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:46:02.979 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:46:02.979 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_1 2022-04-10T01:46:02.979 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=present 2022-04-10T01:46:02.979 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id=113399b8ea92 2022-04-10T01:46:02.980 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:46:02.980 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:46:02.980 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:46:02.980 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:46:02.981 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:46:02.981 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:46:02.981 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_1 2022-04-10T01:46:02.981 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:46:02.982 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:46:02.982 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_1 2022-04-10T01:46:02.982 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:46:02.982 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:46:02.983 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:46:02.983 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:46:02.983 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2022-04-10T01:46:02.983 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:46:03.020 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id=113399b8ea92 2022-04-10T01:46:03.021 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n 113399b8ea92 2022-04-10T01:46:03.021 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -z 113399b8ea92 -o 113399b8ea92 = 113399b8ea92 2022-04-10T01:46:03.021 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=present 2022-04-10T01:46:03.021 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test present = present 2022-04-10T01:46:03.022 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:46:03.022 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ disable_mirror cluster2 mirror image_1 2022-04-10T01:46:03.022 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:46:03.022 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:46:03.022 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_1 2022-04-10T01:46:03.023 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:46:03.023 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:46:03.023 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:46:03.023 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:46:03.024 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_1 2022-04-10T01:46:03.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi043.stderr:2022-04-10T01:46:03.886+0000 7fd3b2bd0700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559a81377c00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=369 2022-04-10T01:46:03.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi043.stderr:2022-04-10T01:46:03.886+0000 7fd3b2bd0700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559a81377c00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=368, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=e5310ca0-9207-433f-b9d4-5af4554f3d51, primary_snap_id=171, last_copied_object_number=128, snap_seqs={369=18446744073709551614}] 2022-04-10T01:46:06.333 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~5s 2022-04-10T01:46:08.247 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi043.stderr:2022-04-10T01:46:08.244+0000 7fc2c294c700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55b0c0555860 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:46:08.247 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi043.stderr:2022-04-10T01:46:08.244+0000 7fc2c9159700 -1 rbd::mirror::ImageReplayer: 0x55b0bc11fa00 [3/53c54804-c4f0-4217-a937-1223c63bfc97] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T01:46:08.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring disabled 2022-04-10T01:46:08.677 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2022-04-10T01:46:08.677 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:46:08.677 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:46:08.677 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_1 2022-04-10T01:46:08.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:46:08.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:46:08.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:46:08.678 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:46:08.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:46:08.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:46:08.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:46:08.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:46:08.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_1 2022-04-10T01:46:08.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:46:08.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:46:08.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_1 2022-04-10T01:46:08.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:46:08.681 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:46:08.681 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:46:08.681 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:46:08.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2022-04-10T01:46:08.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:46:08.710 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_1: (2) No such file or directory 2022-04-10T01:46:08.714 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:46:08.714 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:46:08.714 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:46:08.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2022-04-10T01:46:08.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:46:08.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:46:08.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_1 2022-04-10T01:46:08.819 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:46:08.819 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:46:08.819 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:46:08.819 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:46:08.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:46:08.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:46:08.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:46:08.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:46:08.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_1 2022-04-10T01:46:08.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:46:08.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:46:08.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_1 2022-04-10T01:46:08.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:46:08.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:46:08.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:46:08.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:46:08.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2022-04-10T01:46:08.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:46:08.852 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_1: (2) No such file or directory 2022-04-10T01:46:08.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:46:08.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:46:08.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:46:08.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:46:08.858 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ purge_snapshots cluster2 mirror image_1 2022-04-10T01:46:08.858 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:46:08.858 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:46:08.858 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_1 2022-04-10T01:46:08.859 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:46:08.859 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:46:08.859 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:46:08.859 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:46:08.860 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap purge mirror/image_1 2022-04-10T01:46:08.900 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_1 2022-04-10T01:46:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:46:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:46:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_1 2022-04-10T01:46:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:46:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:46:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:46:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:46:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:46:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:46:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_1 2022-04-10T01:46:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:46:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:46:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_1 2022-04-10T01:46:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:46:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:46:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:46:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:46:08.905 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2022-04-10T01:46:08.941 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2022-04-10T01:46:11.737 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~11s 2022-04-10T01:46:17.141 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~16s 2022-04-10T01:46:22.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~22s 2022-04-10T01:46:27.949 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~27s 2022-04-10T01:46:33.353 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~32s 2022-04-10T01:46:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2022-04-10T01:46:34.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:46:34.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 10` 2022-04-10T01:46:34.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_2 2022-04-10T01:46:34.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_2 512M 2022-04-10T01:46:34.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:46:34.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:46:34.669 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:46:34.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:46:34.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_2 2022-04-10T01:46:34.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:46:34.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:46:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:46:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:46:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:46:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:46:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 512M ']' 2022-04-10T01:46:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=512M 2022-04-10T01:46:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:46:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_2 2022-04-10T01:46:34.760 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_2 2022-04-10T01:46:34.761 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:46:34.761 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:46:34.761 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_2 2022-04-10T01:46:34.761 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:46:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:46:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:46:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:46:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:46:34.763 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2022-04-10T01:46:36.045 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:46:36.178 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2022-04-10T01:46:36.701 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_2': 2022-04-10T01:46:36.701 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 512 MiB in 128 objects 2022-04-10T01:46:36.701 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:46:36.701 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:46:36.702 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 187c3914673c 2022-04-10T01:46:36.702 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.187c3914673c 2022-04-10T01:46:36.702 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:46:36.702 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:46:36.703 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:46:36.703 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:46:36.703 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:46:34 2022 2022-04-10T01:46:36.703 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:46:34 2022 2022-04-10T01:46:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:46:34 2022 2022-04-10T01:46:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:46:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:46:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: e4fc30c2-7125-4643-bd63-4f744eec56ad 2022-04-10T01:46:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:46:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_2 2022-04-10T01:46:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:46:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:46:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_2 2022-04-10T01:46:36.775 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:46:36.775 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=8 2022-04-10T01:46:36.775 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:46:36.775 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:46:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:46:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:46:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:46:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:46:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 52485120 bytes 2022-04-10T01:46:38.142 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 97953792 bytes 2022-04-10T01:46:38.289 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 127950848 bytes 2022-04-10T01:46:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 139517952 bytes 2022-04-10T01:46:38.487 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 161736704 bytes 2022-04-10T01:46:38.634 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 239832064 bytes 2022-04-10T01:46:38.751 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 286836736 bytes 2022-04-10T01:46:38.757 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~38s 2022-04-10T01:46:38.874 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 330406912 bytes 2022-04-10T01:46:38.987 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 387382272 bytes 2022-04-10T01:46:39.091 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 423094272 bytes 2022-04-10T01:46:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 470718464 bytes 2022-04-10T01:46:39.330 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 553563136 bytes 2022-04-10T01:46:39.435 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 582466560 bytes 2022-04-10T01:46:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 637533184 bytes 2022-04-10T01:46:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 692442112 bytes 2022-04-10T01:46:39.781 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 726075392 bytes 2022-04-10T01:46:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 769090560 bytes 2022-04-10T01:46:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 809009152 bytes 2022-04-10T01:46:40.099 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 859957248 bytes 2022-04-10T01:46:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 924059648 bytes 2022-04-10T01:46:40.360 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 978300928 bytes 2022-04-10T01:46:40.465 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 1019276288 bytes 2022-04-10T01:46:40.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 1096474624 bytes 2022-04-10T01:46:40.708 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 1134051328 bytes 2022-04-10T01:46:40.827 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 1201857536 bytes 2022-04-10T01:46:40.935 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1227302912 bytes 2022-04-10T01:46:40.991 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2022-04-10T01:46:40.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2022-04-10T01:46:40.993 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2022-04-10T01:46:40.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi043.stderr:2022-04-10T01:46:40.992+0000 7fc440401700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 55383) UID: 1000 2022-04-10T01:46:41.031 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1258437632 bytes 2022-04-10T01:46:41.146 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1308211200 bytes 2022-04-10T01:46:41.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi043.stderr:daemon-helper: command failed with exit status 1 2022-04-10T01:46:41.226 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1316049920 bytes 2022-04-10T01:46:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1374432256 bytes 2022-04-10T01:46:41.417 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1375052800 bytes 2022-04-10T01:46:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1421467648 bytes 2022-04-10T01:46:41.614 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1443828736 bytes 2022-04-10T01:46:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1521048576 bytes 2022-04-10T01:46:41.850 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1525148672 bytes 2022-04-10T01:46:41.953 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1563098112 bytes 2022-04-10T01:46:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1633189888 bytes 2022-04-10T01:46:42.160 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1636168704 bytes 2022-04-10T01:46:42.249 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1655928832 bytes 2022-04-10T01:46:42.386 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1720894464 bytes 2022-04-10T01:46:42.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1773838336 bytes 2022-04-10T01:46:42.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 1808325632 bytes 2022-04-10T01:46:42.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 1825910784 bytes 2022-04-10T01:46:42.827 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 1901120512 bytes 2022-04-10T01:46:42.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 1940621312 bytes 2022-04-10T01:46:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 1973580800 bytes 2022-04-10T01:46:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 2028520448 bytes 2022-04-10T01:46:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 2063563776 bytes 2022-04-10T01:46:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 2113516544 bytes 2022-04-10T01:46:43.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 2186863616 bytes 2022-04-10T01:46:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 2259930112 bytes 2022-04-10T01:46:43.771 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 2316661760 bytes 2022-04-10T01:46:43.884 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 2365579264 bytes 2022-04-10T01:46:43.979 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5400 IOs, 2392811520 bytes 2022-04-10T01:46:44.084 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5500 IOs, 2432505856 bytes 2022-04-10T01:46:44.161 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~43s 2022-04-10T01:46:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5600 IOs, 2465610752 bytes 2022-04-10T01:46:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5700 IOs, 2505257984 bytes 2022-04-10T01:46:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5800 IOs, 2524931072 bytes 2022-04-10T01:46:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5900 IOs, 2556374016 bytes 2022-04-10T01:46:44.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6000 IOs, 2568117248 bytes 2022-04-10T01:46:44.670 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6100 IOs, 2593476608 bytes 2022-04-10T01:46:44.785 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6200 IOs, 2634941440 bytes 2022-04-10T01:46:44.794 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:46:44.795 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:46:44.795 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:46:44.795 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:46:44.796 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_2 present 2022-04-10T01:46:44.796 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:46:44.796 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:46:44.796 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_2 2022-04-10T01:46:44.796 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=present 2022-04-10T01:46:44.797 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:46:44.797 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:46:44.797 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:46:44.797 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:46:44.798 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:46:44.798 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:46:44.798 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:46:44.798 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_2 2022-04-10T01:46:44.798 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:46:44.799 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:46:44.799 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_2 2022-04-10T01:46:44.799 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:46:44.799 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:46:44.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:46:44.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:46:44.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2022-04-10T01:46:44.800 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:46:44.834 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id=1133f6e741cb 2022-04-10T01:46:44.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:46:44.835 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:46:44.937 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_2 present 1133f6e741cb 2022-04-10T01:46:44.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:46:44.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:46:44.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_2 2022-04-10T01:46:44.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=present 2022-04-10T01:46:44.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id=1133f6e741cb 2022-04-10T01:46:44.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:46:44.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:46:44.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:46:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:46:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:46:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:46:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_2 2022-04-10T01:46:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:46:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:46:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_2 2022-04-10T01:46:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:46:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:46:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:46:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:46:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2022-04-10T01:46:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:46:44.977 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id=1133f6e741cb 2022-04-10T01:46:44.978 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n 1133f6e741cb 2022-04-10T01:46:44.978 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -z 1133f6e741cb -o 1133f6e741cb = 1133f6e741cb 2022-04-10T01:46:44.978 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=present 2022-04-10T01:46:44.978 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test present = present 2022-04-10T01:46:44.979 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:46:44.979 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ disable_mirror cluster2 mirror image_2 2022-04-10T01:46:44.979 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:46:44.979 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:46:44.980 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_2 2022-04-10T01:46:44.980 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:46:44.980 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:46:44.980 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:46:44.980 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:46:44.981 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_2 2022-04-10T01:46:46.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi043.stderr:2022-04-10T01:46:46.229+0000 7fd3b23cf700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559a80f46400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=371 2022-04-10T01:46:46.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi043.stderr:2022-04-10T01:46:46.229+0000 7fd3b23cf700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559a80f46400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=370, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=e5310ca0-9207-433f-b9d4-5af4554f3d51, primary_snap_id=173, last_copied_object_number=128, snap_seqs={371=18446744073709551614}] 2022-04-10T01:46:48.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi043.stderr:2022-04-10T01:46:48.951+0000 7fc2c214b700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55b0c0072ea0 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:46:48.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi043.stderr:2022-04-10T01:46:48.951+0000 7fc2c8958700 -1 rbd::mirror::ImageReplayer: 0x55b0c00e5e00 [3/e4fc30c2-7125-4643-bd63-4f744eec56ad] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T01:46:49.565 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~49s 2022-04-10T01:46:49.994 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2022-04-10T01:46:49.994 INFO:teuthology.orchestra.run:waiting for 600 2022-04-10T01:46:49.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-04-10T01:46:49.995 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2022-04-10T01:46:49.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2022-04-10T01:46:49.996 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2022-04-10T01:46:49.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2022-04-10T01:46:49.999 INFO:tasks.rbd_mirror_thrash:waiting for 79 secs before thrashing 2022-04-10T01:46:50.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring disabled 2022-04-10T01:46:50.949 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2022-04-10T01:46:50.949 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:46:50.949 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:46:50.949 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_2 2022-04-10T01:46:50.950 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:46:50.950 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:46:50.950 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:46:50.950 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:46:50.950 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:46:50.951 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:46:50.951 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:46:50.951 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:46:50.951 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_2 2022-04-10T01:46:50.952 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:46:50.952 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:46:50.952 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_2 2022-04-10T01:46:50.952 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:46:50.952 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:46:50.953 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:46:50.953 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:46:50.953 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2022-04-10T01:46:50.953 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:46:50.981 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_2: (2) No such file or directory 2022-04-10T01:46:50.984 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:46:50.984 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:46:50.985 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:46:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2022-04-10T01:46:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:46:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:46:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_2 2022-04-10T01:46:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:46:51.089 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:46:51.089 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:46:51.089 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:46:51.089 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:46:51.089 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:46:51.090 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:46:51.090 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:46:51.090 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_2 2022-04-10T01:46:51.091 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:46:51.091 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:46:51.091 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_2 2022-04-10T01:46:51.091 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:46:51.092 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:46:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:46:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:46:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2022-04-10T01:46:51.094 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:46:51.119 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_2: (2) No such file or directory 2022-04-10T01:46:51.123 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:46:51.123 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:46:51.123 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:46:51.124 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:46:51.124 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ purge_snapshots cluster2 mirror image_2 2022-04-10T01:46:51.124 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:46:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:46:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_2 2022-04-10T01:46:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:46:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:46:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:46:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:46:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap purge mirror/image_2 2022-04-10T01:46:51.165 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_2 2022-04-10T01:46:51.165 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:46:51.165 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:46:51.166 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_2 2022-04-10T01:46:51.166 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:46:51.166 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:46:51.167 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:46:51.167 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:46:51.167 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:46:51.167 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:46:51.169 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_2 2022-04-10T01:46:51.169 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:46:51.169 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:46:51.169 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_2 2022-04-10T01:46:51.170 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:46:51.170 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:46:51.170 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:46:51.170 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:46:51.170 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2022-04-10T01:46:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2022-04-10T01:46:54.968 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~54s 2022-04-10T01:47:00.372 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~59s 2022-04-10T01:47:05.776 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~65s 2022-04-10T01:47:06.204 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8990264064049261) > weight (0.25) 2022-04-10T01:47:06.204 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8194401433166325) > weight (0.25) 2022-04-10T01:47:06.205 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.7886232227814537) > weight (0.25) 2022-04-10T01:47:06.205 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.5225794496707293) > weight (0.25) 2022-04-10T01:47:06.205 INFO:tasks.rbd_mirror_thrash:waiting for 117 secs before thrashing 2022-04-10T01:47:11.179 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~70s 2022-04-10T01:47:16.582 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~76s 2022-04-10T01:47:17.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2022-04-10T01:47:17.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:47:17.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 10` 2022-04-10T01:47:17.564 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_3 2022-04-10T01:47:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_3 512M 2022-04-10T01:47:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:47:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:47:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:47:17.566 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:47:17.566 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_3 2022-04-10T01:47:17.566 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:47:17.566 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:47:17.566 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:47:17.567 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:47:17.567 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:47:17.567 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:47:17.567 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 512M ']' 2022-04-10T01:47:17.568 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=512M 2022-04-10T01:47:17.568 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:47:17.568 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_3 2022-04-10T01:47:17.644 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_3 2022-04-10T01:47:17.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:47:17.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:47:17.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_3 2022-04-10T01:47:17.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:47:17.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:47:17.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:47:17.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:47:17.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:47:17.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2022-04-10T01:47:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:47:19.379 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2022-04-10T01:47:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_3': 2022-04-10T01:47:20.002 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 512 MiB in 128 objects 2022-04-10T01:47:20.002 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:47:20.002 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:47:20.003 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 18984fb71fe7 2022-04-10T01:47:20.003 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.18984fb71fe7 2022-04-10T01:47:20.003 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:47:20.003 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:47:20.003 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:47:20.004 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:47:20.004 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:47:17 2022 2022-04-10T01:47:20.004 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:47:17 2022 2022-04-10T01:47:20.004 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:47:17 2022 2022-04-10T01:47:20.005 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:47:20.005 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:47:20.005 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 43fcf394-35ed-4437-90e2-616af3a6b269 2022-04-10T01:47:20.005 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:47:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_3 2022-04-10T01:47:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:47:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:47:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_3 2022-04-10T01:47:20.154 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:47:20.154 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=9 2022-04-10T01:47:20.154 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:47:20.155 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:47:20.155 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:47:20.155 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:47:20.155 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:47:20.155 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:47:20.791 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 40326144 bytes 2022-04-10T01:47:20.961 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 40948736 bytes 2022-04-10T01:47:21.148 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 73700352 bytes 2022-04-10T01:47:21.305 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 132879360 bytes 2022-04-10T01:47:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 135825408 bytes 2022-04-10T01:47:21.471 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 152765440 bytes 2022-04-10T01:47:21.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 207455232 bytes 2022-04-10T01:47:21.732 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 270864384 bytes 2022-04-10T01:47:21.824 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 287656960 bytes 2022-04-10T01:47:21.984 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 370430976 bytes 2022-04-10T01:47:21.985 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~81s 2022-04-10T01:47:22.090 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 464002048 bytes 2022-04-10T01:47:22.261 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 550106112 bytes 2022-04-10T01:47:22.368 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 585345024 bytes 2022-04-10T01:47:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 674351104 bytes 2022-04-10T01:47:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 730504192 bytes 2022-04-10T01:47:22.727 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 769529856 bytes 2022-04-10T01:47:22.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 788616192 bytes 2022-04-10T01:47:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 864655360 bytes 2022-04-10T01:47:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 887614464 bytes 2022-04-10T01:47:23.127 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 895434752 bytes 2022-04-10T01:47:23.230 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 931693568 bytes 2022-04-10T01:47:23.402 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 1040455680 bytes 2022-04-10T01:47:23.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 1088153600 bytes 2022-04-10T01:47:23.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 1106862080 bytes 2022-04-10T01:47:23.712 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 1148088320 bytes 2022-04-10T01:47:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1235433472 bytes 2022-04-10T01:47:23.988 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1328599040 bytes 2022-04-10T01:47:24.104 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1361771520 bytes 2022-04-10T01:47:24.219 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1427872768 bytes 2022-04-10T01:47:24.299 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1431613440 bytes 2022-04-10T01:47:24.386 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1448123392 bytes 2022-04-10T01:47:24.488 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1478712320 bytes 2022-04-10T01:47:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1502960640 bytes 2022-04-10T01:47:24.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1523234816 bytes 2022-04-10T01:47:24.787 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1558381568 bytes 2022-04-10T01:47:24.882 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1581865984 bytes 2022-04-10T01:47:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1632659456 bytes 2022-04-10T01:47:25.088 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1671882752 bytes 2022-04-10T01:47:25.193 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1695249408 bytes 2022-04-10T01:47:25.270 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1703571456 bytes 2022-04-10T01:47:25.402 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1759936512 bytes 2022-04-10T01:47:25.566 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 1863541760 bytes 2022-04-10T01:47:25.680 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 1912597504 bytes 2022-04-10T01:47:25.759 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 1921250304 bytes 2022-04-10T01:47:25.849 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 1940036608 bytes 2022-04-10T01:47:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 2017657856 bytes 2022-04-10T01:47:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 2089970688 bytes 2022-04-10T01:47:26.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 2145081344 bytes 2022-04-10T01:47:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 2237236224 bytes 2022-04-10T01:47:26.593 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 2369526784 bytes 2022-04-10T01:47:26.706 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 2412140544 bytes 2022-04-10T01:47:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 2431040512 bytes 2022-04-10T01:47:26.890 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 2454104064 bytes 2022-04-10T01:47:27.034 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5400 IOs, 2526191616 bytes 2022-04-10T01:47:27.123 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5500 IOs, 2548385792 bytes 2022-04-10T01:47:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5600 IOs, 2609831936 bytes 2022-04-10T01:47:27.359 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5700 IOs, 2651576320 bytes 2022-04-10T01:47:27.388 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~86s 2022-04-10T01:47:27.450 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5800 IOs, 2668476416 bytes 2022-04-10T01:47:27.542 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5900 IOs, 2710822912 bytes 2022-04-10T01:47:27.671 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6000 IOs, 2760119296 bytes 2022-04-10T01:47:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6100 IOs, 2760740864 bytes 2022-04-10T01:47:27.894 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6200 IOs, 2849120256 bytes 2022-04-10T01:47:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6300 IOs, 2875643904 bytes 2022-04-10T01:47:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6400 IOs, 2929131520 bytes 2022-04-10T01:47:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6500 IOs, 2978321408 bytes 2022-04-10T01:47:28.383 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6600 IOs, 3061132288 bytes 2022-04-10T01:47:28.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6700 IOs, 3130164224 bytes 2022-04-10T01:47:28.630 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6800 IOs, 3188557824 bytes 2022-04-10T01:47:28.735 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6900 IOs, 3227572224 bytes 2022-04-10T01:47:28.827 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7000 IOs, 3246656512 bytes 2022-04-10T01:47:28.918 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7100 IOs, 3268502528 bytes 2022-04-10T01:47:29.024 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7200 IOs, 3311801344 bytes 2022-04-10T01:47:29.153 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7300 IOs, 3372070912 bytes 2022-04-10T01:47:29.171 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:47:29.171 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:47:29.171 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:47:29.172 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:47:29.172 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_3 present 2022-04-10T01:47:29.172 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:47:29.172 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:47:29.173 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_3 2022-04-10T01:47:29.173 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=present 2022-04-10T01:47:29.173 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:47:29.173 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:47:29.174 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:47:29.174 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:47:29.174 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:47:29.174 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:47:29.174 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:47:29.175 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_3 2022-04-10T01:47:29.175 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:47:29.175 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:47:29.175 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_3 2022-04-10T01:47:29.175 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:47:29.176 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:47:29.176 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:47:29.176 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:47:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2022-04-10T01:47:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:47:29.210 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id=11332a1a1509 2022-04-10T01:47:29.211 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:47:29.211 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:47:29.313 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_3 present 11332a1a1509 2022-04-10T01:47:29.313 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:47:29.313 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:47:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_3 2022-04-10T01:47:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=present 2022-04-10T01:47:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id=11332a1a1509 2022-04-10T01:47:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:47:29.315 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:47:29.315 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:47:29.315 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:47:29.315 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:47:29.316 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:47:29.316 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_3 2022-04-10T01:47:29.316 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:47:29.316 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:47:29.316 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_3 2022-04-10T01:47:29.317 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:47:29.317 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:47:29.317 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:47:29.317 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:47:29.317 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2022-04-10T01:47:29.318 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:47:29.354 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id=11332a1a1509 2022-04-10T01:47:29.354 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n 11332a1a1509 2022-04-10T01:47:29.355 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -z 11332a1a1509 -o 11332a1a1509 = 11332a1a1509 2022-04-10T01:47:29.355 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=present 2022-04-10T01:47:29.355 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test present = present 2022-04-10T01:47:29.355 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:47:29.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ disable_mirror cluster2 mirror image_3 2022-04-10T01:47:29.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:47:29.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:47:29.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_3 2022-04-10T01:47:29.357 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:47:29.357 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:47:29.357 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:47:29.357 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:47:29.357 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_3 2022-04-10T01:47:29.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi043.stderr:2022-04-10T01:47:29.926+0000 7fd3b23cf700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559a7e5ba800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=373 2022-04-10T01:47:29.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi043.stderr:2022-04-10T01:47:29.926+0000 7fd3b23cf700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559a7e5ba800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=372, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=e5310ca0-9207-433f-b9d4-5af4554f3d51, primary_snap_id=175, last_copied_object_number=128, snap_seqs={373=18446744073709551614}] 2022-04-10T01:47:32.793 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~92s 2022-04-10T01:47:34.717 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring disabled 2022-04-10T01:47:34.725 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2022-04-10T01:47:34.726 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:47:34.726 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:47:34.726 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_3 2022-04-10T01:47:34.726 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:47:34.727 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:47:34.727 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:47:34.727 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:47:34.727 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:47:34.727 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:47:34.728 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:47:34.728 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:47:34.728 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_3 2022-04-10T01:47:34.728 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:47:34.729 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:47:34.729 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_3 2022-04-10T01:47:34.729 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:47:34.729 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:47:34.729 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:47:34.730 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:47:34.730 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2022-04-10T01:47:34.730 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:47:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_3: (2) No such file or directory 2022-04-10T01:47:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:47:34.763 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:47:34.763 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:47:34.865 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2022-04-10T01:47:34.866 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:47:34.866 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:47:34.866 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_3 2022-04-10T01:47:34.867 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:47:34.867 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:47:34.867 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:47:34.867 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:47:34.868 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:47:34.868 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:47:34.868 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:47:34.868 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:47:34.869 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_3 2022-04-10T01:47:34.869 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:47:34.869 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:47:34.869 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_3 2022-04-10T01:47:34.869 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:47:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:47:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:47:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:47:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2022-04-10T01:47:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:47:34.896 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_3: (2) No such file or directory 2022-04-10T01:47:34.900 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:47:34.900 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:47:34.900 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:47:34.901 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:47:34.901 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ purge_snapshots cluster2 mirror image_3 2022-04-10T01:47:34.901 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:47:34.901 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:47:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_3 2022-04-10T01:47:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:47:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:47:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:47:34.903 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:47:34.903 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap purge mirror/image_3 2022-04-10T01:47:34.942 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_3 2022-04-10T01:47:34.943 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:47:34.943 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:47:34.943 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_3 2022-04-10T01:47:34.943 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:47:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:47:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:47:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:47:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:47:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:47:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_3 2022-04-10T01:47:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:47:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:47:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_3 2022-04-10T01:47:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:47:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:47:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:47:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:47:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2022-04-10T01:47:34.983 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2022-04-10T01:47:38.197 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~97s 2022-04-10T01:47:43.602 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~103s 2022-04-10T01:47:49.006 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~108s 2022-04-10T01:47:54.410 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~113s 2022-04-10T01:47:59.718 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2022-04-10T01:47:59.732 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:47:59.732 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 10` 2022-04-10T01:47:59.732 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_4 2022-04-10T01:47:59.733 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_4 512M 2022-04-10T01:47:59.733 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:47:59.733 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:47:59.733 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:47:59.733 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:47:59.734 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_4 2022-04-10T01:47:59.734 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:47:59.734 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:47:59.734 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:47:59.735 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:47:59.736 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:47:59.736 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:47:59.736 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 512M ']' 2022-04-10T01:47:59.736 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=512M 2022-04-10T01:47:59.736 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:47:59.737 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_4 2022-04-10T01:47:59.814 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~119s 2022-04-10T01:47:59.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_4 2022-04-10T01:47:59.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:47:59.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:47:59.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_4 2022-04-10T01:47:59.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:47:59.821 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:47:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:47:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:47:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:47:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2022-04-10T01:48:01.104 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:48:01.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2022-04-10T01:48:01.826 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_4': 2022-04-10T01:48:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 512 MiB in 128 objects 2022-04-10T01:48:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:48:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:48:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 18ab331aaa0e 2022-04-10T01:48:01.828 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.18ab331aaa0e 2022-04-10T01:48:01.828 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:48:01.828 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:48:01.828 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:48:01.828 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:48:01.829 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:47:59 2022 2022-04-10T01:48:01.829 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:47:59 2022 2022-04-10T01:48:01.829 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:47:59 2022 2022-04-10T01:48:01.829 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:48:01.829 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:48:01.830 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 69724ec8-0005-476b-8297-4e238918f4c8 2022-04-10T01:48:01.830 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:48:01.891 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_4 2022-04-10T01:48:01.891 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:48:01.891 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:48:01.892 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_4 2022-04-10T01:48:01.905 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:48:01.916 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=12 2022-04-10T01:48:01.916 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:48:01.917 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:48:01.917 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:48:01.917 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:48:01.918 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:48:01.918 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:48:02.871 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 46586880 bytes 2022-04-10T01:48:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 91700224 bytes 2022-04-10T01:48:03.223 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 147847168 bytes 2022-04-10T01:48:03.373 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 208729088 bytes 2022-04-10T01:48:03.474 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 242915328 bytes 2022-04-10T01:48:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 278394880 bytes 2022-04-10T01:48:03.684 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 304281600 bytes 2022-04-10T01:48:03.784 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 328751104 bytes 2022-04-10T01:48:03.897 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 368236544 bytes 2022-04-10T01:48:04.112 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 392493056 bytes 2022-04-10T01:48:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 399607808 bytes 2022-04-10T01:48:04.359 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 472085504 bytes 2022-04-10T01:48:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 522276864 bytes 2022-04-10T01:48:04.574 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 541381632 bytes 2022-04-10T01:48:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 611938304 bytes 2022-04-10T01:48:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 613781504 bytes 2022-04-10T01:48:04.844 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 618160128 bytes 2022-04-10T01:48:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 656095232 bytes 2022-04-10T01:48:05.035 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 679502848 bytes 2022-04-10T01:48:05.135 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 722039808 bytes 2022-04-10T01:48:05.218 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~124s 2022-04-10T01:48:05.334 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 747522048 bytes 2022-04-10T01:48:05.335 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 752070656 bytes 2022-04-10T01:48:05.429 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 793319424 bytes 2022-04-10T01:48:05.539 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 839478272 bytes 2022-04-10T01:48:05.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 920547328 bytes 2022-04-10T01:48:05.811 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 952011776 bytes 2022-04-10T01:48:05.931 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1003787264 bytes 2022-04-10T01:48:06.035 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1039009792 bytes 2022-04-10T01:48:06.124 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1054063616 bytes 2022-04-10T01:48:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1105847296 bytes 2022-04-10T01:48:06.335 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1124144128 bytes 2022-04-10T01:48:06.491 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1220895744 bytes 2022-04-10T01:48:06.610 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1290570752 bytes 2022-04-10T01:48:06.727 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1323712512 bytes 2022-04-10T01:48:06.861 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1389691904 bytes 2022-04-10T01:48:06.965 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1419383808 bytes 2022-04-10T01:48:07.056 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1463077888 bytes 2022-04-10T01:48:07.165 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1505543168 bytes 2022-04-10T01:48:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1561339904 bytes 2022-04-10T01:48:07.393 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1577077760 bytes 2022-04-10T01:48:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1602007040 bytes 2022-04-10T01:48:07.621 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 1684120576 bytes 2022-04-10T01:48:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 1706233856 bytes 2022-04-10T01:48:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 1774658560 bytes 2022-04-10T01:48:07.933 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 1775685632 bytes 2022-04-10T01:48:08.033 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 1793589248 bytes 2022-04-10T01:48:08.106 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 1798221824 bytes 2022-04-10T01:48:08.211 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 1841573888 bytes 2022-04-10T01:48:08.295 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 1848512512 bytes 2022-04-10T01:48:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 1900108800 bytes 2022-04-10T01:48:08.545 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 1976007680 bytes 2022-04-10T01:48:08.659 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 2017833984 bytes 2022-04-10T01:48:08.788 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 2085515264 bytes 2022-04-10T01:48:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5400 IOs, 2146471936 bytes 2022-04-10T01:48:09.000 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2022-04-10T01:48:09.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2022-04-10T01:48:09.001 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2022-04-10T01:48:09.002 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2022-04-10T01:48:09.002 INFO:teuthology.orchestra.run:waiting for 600 2022-04-10T01:48:09.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi043.stderr:2022-04-10T01:48:09.000+0000 7f6f8f837700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 72628) UID: 1000 2022-04-10T01:48:09.073 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5500 IOs, 2245209088 bytes 2022-04-10T01:48:09.170 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5600 IOs, 2269201408 bytes 2022-04-10T01:48:09.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi043.stderr:daemon-helper: command failed with exit status 1 2022-04-10T01:48:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5700 IOs, 2317545472 bytes 2022-04-10T01:48:09.412 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5800 IOs, 2372221952 bytes 2022-04-10T01:48:09.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5900 IOs, 2376235008 bytes 2022-04-10T01:48:09.579 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6000 IOs, 2396599296 bytes 2022-04-10T01:48:09.707 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6100 IOs, 2450290688 bytes 2022-04-10T01:48:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6200 IOs, 2487367680 bytes 2022-04-10T01:48:09.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6300 IOs, 2545303552 bytes 2022-04-10T01:48:10.025 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6400 IOs, 2558388224 bytes 2022-04-10T01:48:10.105 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6500 IOs, 2569707520 bytes 2022-04-10T01:48:10.202 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6600 IOs, 2599530496 bytes 2022-04-10T01:48:10.290 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6700 IOs, 2619822080 bytes 2022-04-10T01:48:10.419 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6800 IOs, 2690018304 bytes 2022-04-10T01:48:10.521 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6900 IOs, 2704848896 bytes 2022-04-10T01:48:10.626 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7000 IOs, 2742406144 bytes 2022-04-10T01:48:10.717 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7100 IOs, 2760498176 bytes 2022-04-10T01:48:10.737 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~130s 2022-04-10T01:48:10.810 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7200 IOs, 2777919488 bytes 2022-04-10T01:48:10.907 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7300 IOs, 2804858880 bytes 2022-04-10T01:48:11.013 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7400 IOs, 2842963968 bytes 2022-04-10T01:48:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7500 IOs, 2909964288 bytes 2022-04-10T01:48:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7600 IOs, 2960585728 bytes 2022-04-10T01:48:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7700 IOs, 3031887872 bytes 2022-04-10T01:48:11.582 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7800 IOs, 3151329280 bytes 2022-04-10T01:48:11.751 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7900 IOs, 3253505024 bytes 2022-04-10T01:48:11.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8000 IOs, 3318105088 bytes 2022-04-10T01:48:11.983 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8100 IOs, 3359250432 bytes 2022-04-10T01:48:12.080 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8200 IOs, 3380063232 bytes 2022-04-10T01:48:12.178 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8300 IOs, 3411706880 bytes 2022-04-10T01:48:12.282 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8400 IOs, 3445682176 bytes 2022-04-10T01:48:12.371 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8500 IOs, 3462673408 bytes 2022-04-10T01:48:12.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8600 IOs, 3545451520 bytes 2022-04-10T01:48:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8700 IOs, 3564521472 bytes 2022-04-10T01:48:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8800 IOs, 3602119680 bytes 2022-04-10T01:48:12.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8900 IOs, 3647650816 bytes 2022-04-10T01:48:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:9000 IOs, 3676825600 bytes 2022-04-10T01:48:13.008 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:9100 IOs, 3685464064 bytes 2022-04-10T01:48:13.108 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:9200 IOs, 3714621440 bytes 2022-04-10T01:48:13.224 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:9300 IOs, 3757802496 bytes 2022-04-10T01:48:13.313 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:9400 IOs, 3774438400 bytes 2022-04-10T01:48:13.440 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:9500 IOs, 3813945344 bytes 2022-04-10T01:48:13.560 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:9600 IOs, 3873612800 bytes 2022-04-10T01:48:13.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:9700 IOs, 3886030848 bytes 2022-04-10T01:48:13.782 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:9800 IOs, 3956997120 bytes 2022-04-10T01:48:13.906 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:9900 IOs, 4015537152 bytes 2022-04-10T01:48:13.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:48:13.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:48:13.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:48:13.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:48:13.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_4 present 2022-04-10T01:48:13.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:48:13.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:48:13.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_4 2022-04-10T01:48:13.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=present 2022-04-10T01:48:13.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:48:13.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:48:13.941 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:48:13.941 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:48:13.942 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:48:13.943 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:48:13.943 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:48:13.943 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_4 2022-04-10T01:48:13.943 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:48:13.943 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:48:13.944 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_4 2022-04-10T01:48:13.944 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:48:13.944 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:48:13.944 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:48:13.944 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:48:13.945 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2022-04-10T01:48:13.945 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:48:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id=1133119dbe5 2022-04-10T01:48:13.980 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:48:13.980 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:48:14.082 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_4 present 1133119dbe5 2022-04-10T01:48:14.082 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:48:14.083 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:48:14.083 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_4 2022-04-10T01:48:14.083 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=present 2022-04-10T01:48:14.083 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id=1133119dbe5 2022-04-10T01:48:14.083 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:48:14.084 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:48:14.084 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:48:14.084 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:48:14.084 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:48:14.084 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:48:14.085 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_4 2022-04-10T01:48:14.085 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:48:14.085 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:48:14.085 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_4 2022-04-10T01:48:14.085 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:48:14.086 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:48:14.086 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:48:14.086 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:48:14.086 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2022-04-10T01:48:14.086 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:48:14.121 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id=1133119dbe5 2022-04-10T01:48:14.121 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n 1133119dbe5 2022-04-10T01:48:14.121 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -z 1133119dbe5 -o 1133119dbe5 = 1133119dbe5 2022-04-10T01:48:14.121 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=present 2022-04-10T01:48:14.121 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test present = present 2022-04-10T01:48:14.122 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:48:14.122 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ disable_mirror cluster2 mirror image_4 2022-04-10T01:48:14.122 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:48:14.122 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:48:14.123 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_4 2022-04-10T01:48:14.123 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:48:14.123 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:48:14.123 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:48:14.123 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:48:14.124 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_4 2022-04-10T01:48:15.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi043.stderr:2022-04-10T01:48:15.018+0000 7fd3b2bd0700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559a7dade000 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=375 2022-04-10T01:48:15.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi043.stderr:2022-04-10T01:48:15.018+0000 7fd3b2bd0700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559a7dade000 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=374, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=e5310ca0-9207-433f-b9d4-5af4554f3d51, primary_snap_id=177, last_copied_object_number=128, snap_seqs={375=18446744073709551614}] 2022-04-10T01:48:15.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-04-10T01:48:15.103 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2022-04-10T01:48:15.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2022-04-10T01:48:15.104 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2022-04-10T01:48:15.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2022-04-10T01:48:15.151 INFO:tasks.rbd_mirror_thrash:waiting for 112 secs before thrashing 2022-04-10T01:48:16.140 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~135s 2022-04-10T01:48:18.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi043.stderr:2022-04-10T01:48:18.959+0000 7fc2c294c700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55b0c0072340 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:48:18.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi043.stderr:2022-04-10T01:48:18.959+0000 7fc2c8958700 -1 rbd::mirror::ImageReplayer: 0x55b0c00e5e00 [3/69724ec8-0005-476b-8297-4e238918f4c8] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T01:48:18.965 INFO:tasks.ceph.cluster1.osd.2.smithi005.stderr:2022-04-10T01:48:18.963+0000 7f0c60f96700 -1 reset not still connected to 0x564c161d4000 2022-04-10T01:48:18.966 INFO:tasks.ceph.cluster1.osd.2.smithi005.stderr:2022-04-10T01:48:18.963+0000 7f0c60f96700 -1 reset not still connected to 0x564c161d4dd0 2022-04-10T01:48:18.966 INFO:tasks.ceph.cluster1.osd.2.smithi005.stderr:2022-04-10T01:48:18.963+0000 7f0c60f96700 -1 reset not still connected to 0x564c161d4f70 2022-04-10T01:48:19.884 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring disabled 2022-04-10T01:48:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2022-04-10T01:48:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:48:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:48:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_4 2022-04-10T01:48:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:48:19.894 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:48:19.894 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:48:19.894 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:48:19.894 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:48:19.895 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:48:19.895 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:48:19.895 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:48:19.895 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_4 2022-04-10T01:48:19.895 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:48:19.896 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:48:19.896 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_4 2022-04-10T01:48:19.896 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:48:19.896 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:48:19.896 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:48:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:48:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2022-04-10T01:48:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:48:19.930 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_4: (2) No such file or directory 2022-04-10T01:48:19.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:48:19.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:48:19.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:48:20.036 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2022-04-10T01:48:20.037 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:48:20.037 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:48:20.037 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_4 2022-04-10T01:48:20.038 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:48:20.038 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:48:20.038 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:48:20.038 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:48:20.039 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:48:20.039 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:48:20.039 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:48:20.039 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:48:20.039 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_4 2022-04-10T01:48:20.040 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:48:20.040 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:48:20.040 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_4 2022-04-10T01:48:20.040 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:48:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:48:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:48:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:48:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2022-04-10T01:48:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:48:20.067 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_4: (2) No such file or directory 2022-04-10T01:48:20.071 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:48:20.071 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:48:20.072 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:48:20.072 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:48:20.072 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ purge_snapshots cluster2 mirror image_4 2022-04-10T01:48:20.072 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:48:20.073 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:48:20.073 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_4 2022-04-10T01:48:20.073 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:48:20.073 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:48:20.074 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:48:20.074 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:48:20.074 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap purge mirror/image_4 2022-04-10T01:48:20.114 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_4 2022-04-10T01:48:20.114 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:48:20.115 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:48:20.115 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_4 2022-04-10T01:48:20.115 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:48:20.115 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:48:20.116 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:48:20.116 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:48:20.116 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:48:20.116 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:48:20.116 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_4 2022-04-10T01:48:20.117 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:48:20.117 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:48:20.117 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_4 2022-04-10T01:48:20.117 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:48:20.118 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:48:20.118 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:48:20.118 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:48:20.118 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2022-04-10T01:48:20.155 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2022-04-10T01:48:21.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~141s 2022-04-10T01:48:26.948 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~146s 2022-04-10T01:48:32.352 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~151s 2022-04-10T01:48:37.755 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~157s 2022-04-10T01:48:43.159 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~162s 2022-04-10T01:48:44.807 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2022-04-10T01:48:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:48:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 10` 2022-04-10T01:48:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_5 2022-04-10T01:48:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_5 512M 2022-04-10T01:48:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:48:44.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:48:44.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:48:44.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:48:44.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_5 2022-04-10T01:48:44.818 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:48:44.819 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:48:44.819 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:48:44.819 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:48:44.819 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:48:44.819 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:48:44.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 512M ']' 2022-04-10T01:48:44.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=512M 2022-04-10T01:48:44.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:48:44.820 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_5 2022-04-10T01:48:44.874 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_5 2022-04-10T01:48:44.874 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:48:44.874 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:48:44.874 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_5 2022-04-10T01:48:44.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:48:44.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:48:44.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:48:44.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:48:44.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:48:44.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2022-04-10T01:48:46.523 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:48:46.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2022-04-10T01:48:47.399 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_5': 2022-04-10T01:48:47.399 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 512 MiB in 128 objects 2022-04-10T01:48:47.399 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:48:47.399 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:48:47.400 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 18c21e1c70dc 2022-04-10T01:48:47.400 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.18c21e1c70dc 2022-04-10T01:48:47.400 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:48:47.400 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:48:47.401 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:48:47.401 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:48:47.401 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:48:44 2022 2022-04-10T01:48:47.401 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:48:44 2022 2022-04-10T01:48:47.401 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:48:44 2022 2022-04-10T01:48:47.402 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:48:47.402 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:48:47.402 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 887a13d2-78f1-49ef-9ee6-cd7b2282b379 2022-04-10T01:48:47.402 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:48:47.467 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_5 2022-04-10T01:48:47.467 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:48:47.467 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:48:47.468 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_5 2022-04-10T01:48:47.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:48:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=8 2022-04-10T01:48:47.482 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:48:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:48:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:48:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:48:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:48:47.484 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:48:48.211 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 77231104 bytes 2022-04-10T01:48:48.380 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 133913600 bytes 2022-04-10T01:48:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 169970688 bytes 2022-04-10T01:48:48.563 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~168s 2022-04-10T01:48:48.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 226076672 bytes 2022-04-10T01:48:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 265892864 bytes 2022-04-10T01:48:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 295323648 bytes 2022-04-10T01:48:49.038 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 355009536 bytes 2022-04-10T01:48:49.153 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 395628544 bytes 2022-04-10T01:48:49.272 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 414699520 bytes 2022-04-10T01:48:49.359 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 422935552 bytes 2022-04-10T01:48:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 445417472 bytes 2022-04-10T01:48:49.592 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 513718272 bytes 2022-04-10T01:48:49.737 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 600026112 bytes 2022-04-10T01:48:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 643343360 bytes 2022-04-10T01:48:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 732197888 bytes 2022-04-10T01:48:50.102 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 768181248 bytes 2022-04-10T01:48:50.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 812750848 bytes 2022-04-10T01:48:50.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 852698112 bytes 2022-04-10T01:48:50.492 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 952690688 bytes 2022-04-10T01:48:50.605 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 995336192 bytes 2022-04-10T01:48:50.738 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 1061384192 bytes 2022-04-10T01:48:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 1070520320 bytes 2022-04-10T01:48:50.920 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 1120889856 bytes 2022-04-10T01:48:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 1177486336 bytes 2022-04-10T01:48:51.164 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 1207837696 bytes 2022-04-10T01:48:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1265965056 bytes 2022-04-10T01:48:51.360 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1281942528 bytes 2022-04-10T01:48:51.456 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1305238528 bytes 2022-04-10T01:48:51.581 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1357092864 bytes 2022-04-10T01:48:51.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1396396032 bytes 2022-04-10T01:48:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1475729408 bytes 2022-04-10T01:48:51.951 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1544960000 bytes 2022-04-10T01:48:52.033 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1545581568 bytes 2022-04-10T01:48:52.151 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1583486976 bytes 2022-04-10T01:48:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1618929664 bytes 2022-04-10T01:48:52.367 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1663753216 bytes 2022-04-10T01:48:52.458 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1692323840 bytes 2022-04-10T01:48:52.570 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1736250368 bytes 2022-04-10T01:48:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1809812480 bytes 2022-04-10T01:48:52.822 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1875282944 bytes 2022-04-10T01:48:52.972 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1935034368 bytes 2022-04-10T01:48:53.122 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 2009873408 bytes 2022-04-10T01:48:53.270 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 2082220032 bytes 2022-04-10T01:48:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 2119386112 bytes 2022-04-10T01:48:53.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 2187105280 bytes 2022-04-10T01:48:53.624 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 2231400448 bytes 2022-04-10T01:48:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 2314978304 bytes 2022-04-10T01:48:53.911 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 2335896576 bytes 2022-04-10T01:48:53.966 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~173s 2022-04-10T01:48:54.008 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 2362339328 bytes 2022-04-10T01:48:54.150 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 2405804032 bytes 2022-04-10T01:48:54.266 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 2447294464 bytes 2022-04-10T01:48:54.372 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 2485346304 bytes 2022-04-10T01:48:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 2505424896 bytes 2022-04-10T01:48:54.547 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5400 IOs, 2517467136 bytes 2022-04-10T01:48:54.683 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5500 IOs, 2577150976 bytes 2022-04-10T01:48:54.787 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5600 IOs, 2621450240 bytes 2022-04-10T01:48:54.908 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5700 IOs, 2681456640 bytes 2022-04-10T01:48:55.022 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5800 IOs, 2733627392 bytes 2022-04-10T01:48:55.119 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5900 IOs, 2753981440 bytes 2022-04-10T01:48:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6000 IOs, 2774327296 bytes 2022-04-10T01:48:55.343 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6100 IOs, 2852426752 bytes 2022-04-10T01:48:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6200 IOs, 2876094464 bytes 2022-04-10T01:48:55.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:48:55.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:48:55.512 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:48:55.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:48:55.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_5 present 2022-04-10T01:48:55.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:48:55.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:48:55.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_5 2022-04-10T01:48:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=present 2022-04-10T01:48:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:48:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:48:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:48:55.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:48:55.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:48:55.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:48:55.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:48:55.515 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_5 2022-04-10T01:48:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:48:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:48:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_5 2022-04-10T01:48:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:48:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:48:55.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:48:55.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:48:55.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2022-04-10T01:48:55.517 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:48:55.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id=11337048f0ab 2022-04-10T01:48:55.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:48:55.550 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:48:55.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_5 present 11337048f0ab 2022-04-10T01:48:55.652 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:48:55.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:48:55.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_5 2022-04-10T01:48:55.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=present 2022-04-10T01:48:55.653 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id=11337048f0ab 2022-04-10T01:48:55.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:48:55.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:48:55.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:48:55.654 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:48:55.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:48:55.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:48:55.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_5 2022-04-10T01:48:55.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:48:55.655 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:48:55.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_5 2022-04-10T01:48:55.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:48:55.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:48:55.656 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:48:55.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:48:55.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2022-04-10T01:48:55.657 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:48:55.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id=11337048f0ab 2022-04-10T01:48:55.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n 11337048f0ab 2022-04-10T01:48:55.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -z 11337048f0ab -o 11337048f0ab = 11337048f0ab 2022-04-10T01:48:55.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=present 2022-04-10T01:48:55.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test present = present 2022-04-10T01:48:55.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:48:55.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ disable_mirror cluster2 mirror image_5 2022-04-10T01:48:55.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:48:55.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:48:55.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_5 2022-04-10T01:48:55.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:48:55.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:48:55.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:48:55.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:48:55.695 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_5 2022-04-10T01:48:56.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi043.stderr:2022-04-10T01:48:56.195+0000 7fd3b2bd0700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559a7e410c00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=377 2022-04-10T01:48:56.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi043.stderr:2022-04-10T01:48:56.195+0000 7fd3b2bd0700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559a7e410c00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=376, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=e5310ca0-9207-433f-b9d4-5af4554f3d51, primary_snap_id=179, last_copied_object_number=128, snap_seqs={377=18446744073709551614}] 2022-04-10T01:48:59.370 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~178s 2022-04-10T01:49:01.126 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring disabled 2022-04-10T01:49:01.135 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2022-04-10T01:49:01.135 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:49:01.135 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:49:01.135 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_5 2022-04-10T01:49:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:49:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:49:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:49:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:49:01.137 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:49:01.137 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:49:01.137 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:49:01.137 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:49:01.138 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_5 2022-04-10T01:49:01.138 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:49:01.138 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:49:01.138 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_5 2022-04-10T01:49:01.139 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:49:01.139 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:49:01.139 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:49:01.139 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:49:01.139 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2022-04-10T01:49:01.140 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:49:01.168 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_5: (2) No such file or directory 2022-04-10T01:49:01.172 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:49:01.172 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:49:01.172 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:49:01.274 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2022-04-10T01:49:01.274 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:49:01.275 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:49:01.275 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_5 2022-04-10T01:49:01.275 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:49:01.275 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:49:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:49:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:49:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:49:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:49:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:49:01.277 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:49:01.277 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_5 2022-04-10T01:49:01.277 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:49:01.277 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:49:01.278 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_5 2022-04-10T01:49:01.278 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:49:01.278 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:49:01.278 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:49:01.279 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:49:01.279 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2022-04-10T01:49:01.280 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:49:01.305 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_5: (2) No such file or directory 2022-04-10T01:49:01.309 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:49:01.309 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:49:01.310 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:49:01.310 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:49:01.310 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ purge_snapshots cluster2 mirror image_5 2022-04-10T01:49:01.310 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:49:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:49:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_5 2022-04-10T01:49:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:49:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:49:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:49:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:49:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap purge mirror/image_5 2022-04-10T01:49:01.353 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_5 2022-04-10T01:49:01.353 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:49:01.353 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:49:01.354 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_5 2022-04-10T01:49:01.354 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:49:01.354 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:49:01.354 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:49:01.354 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:49:01.355 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:49:01.355 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:49:01.355 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_5 2022-04-10T01:49:01.355 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:49:01.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:49:01.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_5 2022-04-10T01:49:01.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:49:01.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:49:01.356 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:49:01.357 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:49:01.357 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2022-04-10T01:49:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2022-04-10T01:49:03.206 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.3100567528414032) > weight (0.25) 2022-04-10T01:49:03.206 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9345782673181) > weight (0.25) 2022-04-10T01:49:03.206 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.8132855752861642) > weight (0.25) 2022-04-10T01:49:03.207 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2022-04-10T01:49:03.207 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2022-04-10T01:49:03.208 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2022-04-10T01:49:03.208 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2022-04-10T01:49:03.209 INFO:teuthology.orchestra.run:waiting for 600 2022-04-10T01:49:03.209 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi043.stderr:2022-04-10T01:49:03.206+0000 7fc2c995a700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 45809) UID: 1000 2022-04-10T01:49:03.409 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi043.stderr:daemon-helper: command failed with exit status 1 2022-04-10T01:49:04.775 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~184s 2022-04-10T01:49:09.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-04-10T01:49:09.311 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2022-04-10T01:49:09.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2022-04-10T01:49:09.312 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2022-04-10T01:49:09.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2022-04-10T01:49:09.315 INFO:tasks.rbd_mirror_thrash:waiting for 78 secs before thrashing 2022-04-10T01:49:10.180 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~189s 2022-04-10T01:49:15.585 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~195s 2022-04-10T01:49:20.989 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~200s 2022-04-10T01:49:26.394 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~205s 2022-04-10T01:49:28.129 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2022-04-10T01:49:28.142 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:49:28.142 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 10` 2022-04-10T01:49:28.142 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_6 2022-04-10T01:49:28.142 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_6 512M 2022-04-10T01:49:28.143 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:49:28.143 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:49:28.143 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:49:28.143 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:49:28.143 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_6 2022-04-10T01:49:28.144 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:49:28.144 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:49:28.144 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:49:28.144 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:49:28.144 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:49:28.145 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:49:28.145 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 512M ']' 2022-04-10T01:49:28.145 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=512M 2022-04-10T01:49:28.145 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:49:28.145 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_6 2022-04-10T01:49:28.220 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_6 2022-04-10T01:49:28.220 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:49:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:49:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_6 2022-04-10T01:49:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:49:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:49:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:49:28.222 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:49:28.222 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:49:28.222 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2022-04-10T01:49:29.334 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:49:29.410 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2022-04-10T01:49:29.936 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_6': 2022-04-10T01:49:29.936 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 512 MiB in 128 objects 2022-04-10T01:49:29.936 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:49:29.937 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:49:29.937 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 18e08c42e02e 2022-04-10T01:49:29.937 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.18e08c42e02e 2022-04-10T01:49:29.937 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:49:29.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:49:29.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:49:29.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:49:29.938 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:49:28 2022 2022-04-10T01:49:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:49:28 2022 2022-04-10T01:49:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:49:28 2022 2022-04-10T01:49:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:49:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:49:29.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 97c8b4b7-60b4-4235-9676-e6a5d5ee38bb 2022-04-10T01:49:29.940 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:49:30.051 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_6 2022-04-10T01:49:30.051 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:49:30.051 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:49:30.051 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_6 2022-04-10T01:49:30.052 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:49:30.073 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=11 2022-04-10T01:49:30.073 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:49:30.073 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:49:30.073 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:49:30.074 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:49:30.074 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:49:30.074 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:49:30.934 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 7535616 bytes 2022-04-10T01:49:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 48190464 bytes 2022-04-10T01:49:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 96911360 bytes 2022-04-10T01:49:31.592 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 126160896 bytes 2022-04-10T01:49:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 202395648 bytes 2022-04-10T01:49:31.797 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~211s 2022-04-10T01:49:31.887 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 232913920 bytes 2022-04-10T01:49:31.987 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 258406400 bytes 2022-04-10T01:49:32.092 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 294604800 bytes 2022-04-10T01:49:32.190 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 308454400 bytes 2022-04-10T01:49:32.293 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 332713984 bytes 2022-04-10T01:49:32.418 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 388965376 bytes 2022-04-10T01:49:32.583 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 464913408 bytes 2022-04-10T01:49:32.679 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 491883520 bytes 2022-04-10T01:49:32.780 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 522553344 bytes 2022-04-10T01:49:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 591374336 bytes 2022-04-10T01:49:33.024 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 632040448 bytes 2022-04-10T01:49:33.112 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 658707456 bytes 2022-04-10T01:49:33.242 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 709779456 bytes 2022-04-10T01:49:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 817824768 bytes 2022-04-10T01:49:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 858146816 bytes 2022-04-10T01:49:33.628 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 895497216 bytes 2022-04-10T01:49:33.755 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 967524352 bytes 2022-04-10T01:49:33.888 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 1025875968 bytes 2022-04-10T01:49:34.009 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 1080904704 bytes 2022-04-10T01:49:34.132 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 1151358976 bytes 2022-04-10T01:49:34.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1177789440 bytes 2022-04-10T01:49:34.349 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1217184768 bytes 2022-04-10T01:49:34.428 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1217829888 bytes 2022-04-10T01:49:34.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1259804672 bytes 2022-04-10T01:49:34.668 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1318927360 bytes 2022-04-10T01:49:34.773 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1355020288 bytes 2022-04-10T01:49:34.917 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1455480832 bytes 2022-04-10T01:49:35.013 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1469410304 bytes 2022-04-10T01:49:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1523089408 bytes 2022-04-10T01:49:35.281 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1592878080 bytes 2022-04-10T01:49:35.375 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1622574080 bytes 2022-04-10T01:49:35.490 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1664155648 bytes 2022-04-10T01:49:35.608 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1713409024 bytes 2022-04-10T01:49:35.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1725498368 bytes 2022-04-10T01:49:35.840 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1812687872 bytes 2022-04-10T01:49:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 1829234688 bytes 2022-04-10T01:49:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 1855800320 bytes 2022-04-10T01:49:36.094 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 1859535872 bytes 2022-04-10T01:49:36.202 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 1885607936 bytes 2022-04-10T01:49:36.352 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 1952054272 bytes 2022-04-10T01:49:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 2018304000 bytes 2022-04-10T01:49:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 2054007808 bytes 2022-04-10T01:49:36.753 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 2121668608 bytes 2022-04-10T01:49:36.895 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 2195110912 bytes 2022-04-10T01:49:36.978 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 2195722240 bytes 2022-04-10T01:49:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 2284024832 bytes 2022-04-10T01:49:37.201 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~216s 2022-04-10T01:49:37.288 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 2387504128 bytes 2022-04-10T01:49:37.397 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 2417214464 bytes 2022-04-10T01:49:37.500 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5400 IOs, 2456377344 bytes 2022-04-10T01:49:37.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5500 IOs, 2494002176 bytes 2022-04-10T01:49:37.770 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5600 IOs, 2567251968 bytes 2022-04-10T01:49:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5700 IOs, 2610051072 bytes 2022-04-10T01:49:38.030 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5800 IOs, 2685487104 bytes 2022-04-10T01:49:38.117 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5900 IOs, 2705901568 bytes 2022-04-10T01:49:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6000 IOs, 2753492992 bytes 2022-04-10T01:49:38.314 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6100 IOs, 2754111488 bytes 2022-04-10T01:49:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6200 IOs, 2779922432 bytes 2022-04-10T01:49:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6300 IOs, 2790674432 bytes 2022-04-10T01:49:38.620 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6400 IOs, 2838431744 bytes 2022-04-10T01:49:38.717 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6500 IOs, 2870601728 bytes 2022-04-10T01:49:38.806 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6600 IOs, 2871215104 bytes 2022-04-10T01:49:38.899 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6700 IOs, 2887946240 bytes 2022-04-10T01:49:39.009 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6800 IOs, 2932579328 bytes 2022-04-10T01:49:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6900 IOs, 2996753408 bytes 2022-04-10T01:49:39.259 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7000 IOs, 3043368960 bytes 2022-04-10T01:49:39.360 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7100 IOs, 3074100224 bytes 2022-04-10T01:49:39.469 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7200 IOs, 3107400704 bytes 2022-04-10T01:49:39.608 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7300 IOs, 3160114176 bytes 2022-04-10T01:49:39.711 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7400 IOs, 3199693824 bytes 2022-04-10T01:49:39.803 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7500 IOs, 3200295936 bytes 2022-04-10T01:49:39.909 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7600 IOs, 3231204352 bytes 2022-04-10T01:49:40.034 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7700 IOs, 3286621184 bytes 2022-04-10T01:49:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7800 IOs, 3327677440 bytes 2022-04-10T01:49:40.287 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7900 IOs, 3393425408 bytes 2022-04-10T01:49:40.442 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8000 IOs, 3473285120 bytes 2022-04-10T01:49:40.575 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8100 IOs, 3540342784 bytes 2022-04-10T01:49:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8200 IOs, 3573628928 bytes 2022-04-10T01:49:40.845 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8300 IOs, 3657840640 bytes 2022-04-10T01:49:40.951 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8400 IOs, 3695156224 bytes 2022-04-10T01:49:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8500 IOs, 3715331072 bytes 2022-04-10T01:49:41.092 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:49:41.092 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:49:41.093 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:49:41.093 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:49:41.093 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_6 present 2022-04-10T01:49:41.093 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:49:41.093 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:49:41.094 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_6 2022-04-10T01:49:41.094 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=present 2022-04-10T01:49:41.094 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:49:41.094 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:49:41.095 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:49:41.095 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:49:41.095 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:49:41.096 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:49:41.097 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:49:41.097 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_6 2022-04-10T01:49:41.098 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:49:41.098 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:49:41.098 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_6 2022-04-10T01:49:41.098 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:49:41.099 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:49:41.099 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:49:41.099 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:49:41.099 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2022-04-10T01:49:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:49:41.132 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id=1133447ad37d 2022-04-10T01:49:41.132 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:49:41.132 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:49:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_6 present 1133447ad37d 2022-04-10T01:49:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:49:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:49:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_6 2022-04-10T01:49:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=present 2022-04-10T01:49:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id=1133447ad37d 2022-04-10T01:49:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:49:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:49:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:49:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:49:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:49:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:49:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_6 2022-04-10T01:49:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:49:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:49:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_6 2022-04-10T01:49:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:49:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:49:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:49:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:49:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2022-04-10T01:49:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:49:41.277 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id=1133447ad37d 2022-04-10T01:49:41.277 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n 1133447ad37d 2022-04-10T01:49:41.277 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -z 1133447ad37d -o 1133447ad37d = 1133447ad37d 2022-04-10T01:49:41.278 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=present 2022-04-10T01:49:41.278 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test present = present 2022-04-10T01:49:41.278 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:49:41.278 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ disable_mirror cluster2 mirror image_6 2022-04-10T01:49:41.279 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:49:41.279 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:49:41.279 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_6 2022-04-10T01:49:41.279 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:49:41.280 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:49:41.280 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:49:41.280 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:49:41.280 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_6 2022-04-10T01:49:42.301 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi043.stderr:2022-04-10T01:49:42.297+0000 7fd3b23cf700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559a7e00fc00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=379 2022-04-10T01:49:42.301 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi043.stderr:2022-04-10T01:49:42.297+0000 7fd3b23cf700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559a7e00fc00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=378, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=e5310ca0-9207-433f-b9d4-5af4554f3d51, primary_snap_id=17b, last_copied_object_number=128, snap_seqs={379=18446744073709551614}] 2022-04-10T01:49:42.606 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~222s 2022-04-10T01:49:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring disabled 2022-04-10T01:49:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2022-04-10T01:49:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:49:47.308 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:49:47.308 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_6 2022-04-10T01:49:47.308 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:49:47.308 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:49:47.308 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:49:47.309 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:49:47.309 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:49:47.309 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:49:47.309 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:49:47.310 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:49:47.310 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_6 2022-04-10T01:49:47.310 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:49:47.310 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:49:47.311 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_6 2022-04-10T01:49:47.311 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:49:47.311 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:49:47.311 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:49:47.311 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:49:47.312 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2022-04-10T01:49:47.312 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:49:47.341 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_6: (2) No such file or directory 2022-04-10T01:49:47.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:49:47.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:49:47.345 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:49:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2022-04-10T01:49:47.449 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:49:47.449 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:49:47.449 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_6 2022-04-10T01:49:47.449 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:49:47.449 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:49:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:49:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:49:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:49:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:49:47.451 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:49:47.451 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:49:47.451 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_6 2022-04-10T01:49:47.451 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:49:47.452 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:49:47.452 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_6 2022-04-10T01:49:47.452 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:49:47.452 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:49:47.452 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:49:47.453 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:49:47.453 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2022-04-10T01:49:47.453 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:49:47.480 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_6: (2) No such file or directory 2022-04-10T01:49:47.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:49:47.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:49:47.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:49:47.485 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:49:47.486 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ purge_snapshots cluster2 mirror image_6 2022-04-10T01:49:47.486 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:49:47.486 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:49:47.486 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_6 2022-04-10T01:49:47.486 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:49:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:49:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:49:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:49:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap purge mirror/image_6 2022-04-10T01:49:47.530 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_6 2022-04-10T01:49:47.530 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:49:47.530 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:49:47.530 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_6 2022-04-10T01:49:47.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:49:47.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:49:47.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:49:47.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:49:47.531 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:49:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:49:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_6 2022-04-10T01:49:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:49:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:49:47.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_6 2022-04-10T01:49:47.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:49:47.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:49:47.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:49:47.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:49:47.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2022-04-10T01:49:47.573 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2022-04-10T01:49:48.009 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~227s 2022-04-10T01:49:53.413 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~232s 2022-04-10T01:49:58.817 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~238s 2022-04-10T01:50:04.221 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~243s 2022-04-10T01:50:07.152 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8681140560863331) > weight (0.25) 2022-04-10T01:50:07.153 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.43270333292503427) > weight (0.25) 2022-04-10T01:50:07.153 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.5889655421589892) > weight (0.25) 2022-04-10T01:50:07.153 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.7217101453070881) > weight (0.25) 2022-04-10T01:50:07.153 INFO:tasks.rbd_mirror_thrash:waiting for 105 secs before thrashing 2022-04-10T01:50:09.625 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~249s 2022-04-10T01:50:13.309 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2022-04-10T01:50:13.322 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:50:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 10` 2022-04-10T01:50:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_7 2022-04-10T01:50:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_7 512M 2022-04-10T01:50:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:50:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:50:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:50:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:50:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_7 2022-04-10T01:50:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:50:13.325 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:50:13.325 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:50:13.325 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:50:13.325 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:50:13.325 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:50:13.326 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 512M ']' 2022-04-10T01:50:13.326 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=512M 2022-04-10T01:50:13.326 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:50:13.326 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_7 2022-04-10T01:50:13.412 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_7 2022-04-10T01:50:13.413 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:50:13.413 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:50:13.413 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_7 2022-04-10T01:50:13.414 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:50:13.414 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:50:13.414 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:50:13.414 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:50:13.415 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:50:13.415 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2022-04-10T01:50:14.750 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:50:14.881 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2022-04-10T01:50:15.029 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~254s 2022-04-10T01:50:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_7': 2022-04-10T01:50:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 512 MiB in 128 objects 2022-04-10T01:50:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:50:15.317 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:50:15.317 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 18f3921e323c 2022-04-10T01:50:15.317 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.18f3921e323c 2022-04-10T01:50:15.317 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:50:15.317 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:50:15.318 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:50:15.318 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:50:15.318 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:50:13 2022 2022-04-10T01:50:15.318 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:50:13 2022 2022-04-10T01:50:15.319 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:50:13 2022 2022-04-10T01:50:15.319 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:50:15.319 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:50:15.319 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: d4207619-dba4-47d6-8fe0-da5f8bdc5e87 2022-04-10T01:50:15.319 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:50:15.397 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_7 2022-04-10T01:50:15.397 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:50:15.397 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:50:15.398 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_7 2022-04-10T01:50:15.432 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:50:15.434 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=5 2022-04-10T01:50:15.435 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:50:15.435 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:50:15.435 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:50:15.435 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:50:15.436 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:50:15.436 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:50:16.329 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 27866112 bytes 2022-04-10T01:50:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 70534144 bytes 2022-04-10T01:50:16.787 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 87198720 bytes 2022-04-10T01:50:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 156863488 bytes 2022-04-10T01:50:17.083 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 187795456 bytes 2022-04-10T01:50:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 255505408 bytes 2022-04-10T01:50:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 306442240 bytes 2022-04-10T01:50:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 335014912 bytes 2022-04-10T01:50:17.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 354827264 bytes 2022-04-10T01:50:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 390559744 bytes 2022-04-10T01:50:17.762 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 438853632 bytes 2022-04-10T01:50:17.875 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 473679872 bytes 2022-04-10T01:50:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 522877952 bytes 2022-04-10T01:50:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 606373888 bytes 2022-04-10T01:50:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 635157504 bytes 2022-04-10T01:50:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 679776256 bytes 2022-04-10T01:50:18.465 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 711830528 bytes 2022-04-10T01:50:18.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 714252288 bytes 2022-04-10T01:50:18.601 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 749761536 bytes 2022-04-10T01:50:18.726 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 806908928 bytes 2022-04-10T01:50:18.809 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 819218432 bytes 2022-04-10T01:50:18.915 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 873073664 bytes 2022-04-10T01:50:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 931953664 bytes 2022-04-10T01:50:19.168 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 984180736 bytes 2022-04-10T01:50:19.284 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 1031896064 bytes 2022-04-10T01:50:19.386 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1056187392 bytes 2022-04-10T01:50:19.473 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1069216768 bytes 2022-04-10T01:50:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1080046592 bytes 2022-04-10T01:50:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1132272640 bytes 2022-04-10T01:50:19.870 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1133487104 bytes 2022-04-10T01:50:19.983 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1180770304 bytes 2022-04-10T01:50:20.117 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1257298944 bytes 2022-04-10T01:50:20.194 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1257936896 bytes 2022-04-10T01:50:20.324 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1329312768 bytes 2022-04-10T01:50:20.418 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1356577792 bytes 2022-04-10T01:50:20.434 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~260s 2022-04-10T01:50:20.455 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:50:20.455 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:50:20.456 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:50:20.456 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:50:20.456 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_7 present 2022-04-10T01:50:20.456 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:50:20.457 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:50:20.457 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_7 2022-04-10T01:50:20.457 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=present 2022-04-10T01:50:20.457 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:50:20.457 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:50:20.458 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:50:20.458 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:50:20.458 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:50:20.458 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:50:20.459 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:50:20.459 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_7 2022-04-10T01:50:20.459 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:50:20.459 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:50:20.459 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_7 2022-04-10T01:50:20.460 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:50:20.460 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:50:20.460 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:50:20.460 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:50:20.461 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2022-04-10T01:50:20.461 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:50:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id=11336cb76fa5 2022-04-10T01:50:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:50:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:50:20.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_7 present 11336cb76fa5 2022-04-10T01:50:20.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:50:20.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:50:20.597 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_7 2022-04-10T01:50:20.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=present 2022-04-10T01:50:20.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id=11336cb76fa5 2022-04-10T01:50:20.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:50:20.598 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:50:20.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:50:20.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:50:20.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:50:20.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:50:20.599 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_7 2022-04-10T01:50:20.600 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:50:20.600 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:50:20.600 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_7 2022-04-10T01:50:20.600 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:50:20.600 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:50:20.601 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:50:20.601 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:50:20.601 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2022-04-10T01:50:20.601 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:50:20.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id=11336cb76fa5 2022-04-10T01:50:20.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n 11336cb76fa5 2022-04-10T01:50:20.637 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -z 11336cb76fa5 -o 11336cb76fa5 = 11336cb76fa5 2022-04-10T01:50:20.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=present 2022-04-10T01:50:20.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test present = present 2022-04-10T01:50:20.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:50:20.638 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ disable_mirror cluster2 mirror image_7 2022-04-10T01:50:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:50:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:50:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_7 2022-04-10T01:50:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:50:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:50:20.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:50:20.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:50:20.640 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_7 2022-04-10T01:50:21.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr:2022-04-10T01:50:21.384+0000 7f9ecc7f5700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f19f2d2820 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T01:50:21.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr:2022-04-10T01:50:21.384+0000 7f9ed3002700 -1 rbd::mirror::ImageReplayer: 0x55f1a0b2ae00 [3/d4207619-dba4-47d6-8fe0-da5f8bdc5e87] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T01:50:21.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi043.stderr:2022-04-10T01:50:21.586+0000 7fd3b2bd0700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559a81315000 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=381 2022-04-10T01:50:21.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi043.stderr:2022-04-10T01:50:21.586+0000 7fd3b2bd0700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559a81315000 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=380, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=e5310ca0-9207-433f-b9d4-5af4554f3d51, primary_snap_id=17d, last_copied_object_number=128, snap_seqs={381=18446744073709551614}] 2022-04-10T01:50:25.837 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~265s 2022-04-10T01:50:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring disabled 2022-04-10T01:50:26.472 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2022-04-10T01:50:26.472 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:50:26.473 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:50:26.473 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_7 2022-04-10T01:50:26.473 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=deleted 2022-04-10T01:50:26.473 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:50:26.473 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:50:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:50:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:50:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:50:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:50:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:50:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_7 2022-04-10T01:50:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:50:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:50:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_7 2022-04-10T01:50:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:50:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:50:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:50:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:50:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2022-04-10T01:50:26.477 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:50:26.505 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_7: (2) No such file or directory 2022-04-10T01:50:26.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id= 2022-04-10T01:50:26.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:50:26.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:50:26.612 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2022-04-10T01:50:26.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:50:26.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:50:26.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_7 2022-04-10T01:50:26.613 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=deleted 2022-04-10T01:50:26.614 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:50:26.614 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:50:26.615 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:50:26.615 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:50:26.615 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:50:26.616 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:50:26.616 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:50:26.616 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_7 2022-04-10T01:50:26.616 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:50:26.617 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:50:26.617 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_7 2022-04-10T01:50:26.617 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:50:26.617 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:50:26.617 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:50:26.618 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:50:26.618 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2022-04-10T01:50:26.618 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:50:26.641 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: error opening image image_7: (2) No such file or directory 2022-04-10T01:50:26.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id= 2022-04-10T01:50:26.645 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:50:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test deleted = deleted 2022-04-10T01:50:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:50:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ purge_snapshots cluster2 mirror image_7 2022-04-10T01:50:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:50:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:50:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_7 2022-04-10T01:50:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:50:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:50:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:50:26.648 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:50:26.648 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 snap purge mirror/image_7 2022-04-10T01:50:26.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image_retry cluster2 mirror image_7 2022-04-10T01:50:26.688 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:50:26.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:50:26.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_7 2022-04-10T01:50:26.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:50:26.689 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:50:26.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:50:26.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:50:26.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0 1 2 4 8 16 32 2022-04-10T01:50:26.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0 2022-04-10T01:50:26.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ remove_image cluster2 mirror image_7 2022-04-10T01:50:26.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:50:26.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:50:26.692 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_7 2022-04-10T01:50:26.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:50:26.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:50:26.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:50:26.693 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:50:26.694 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2022-04-10T01:50:26.731 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2022-04-10T01:50:27.315 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7675166893326334) > weight (0.25) 2022-04-10T01:50:27.316 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.6429630571238705) > weight (0.25) 2022-04-10T01:50:27.316 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.521968328226879) > weight (0.25) 2022-04-10T01:50:27.316 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.9305162718042801) > weight (0.25) 2022-04-10T01:50:27.317 INFO:tasks.rbd_mirror_thrash:waiting for 94 secs before thrashing 2022-04-10T01:50:31.241 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~270s 2022-04-10T01:50:36.645 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~276s 2022-04-10T01:50:42.049 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~281s 2022-04-10T01:50:47.453 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~287s 2022-04-10T01:50:52.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2022-04-10T01:50:52.524 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:50:52.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for i in `seq 1 10` 2022-04-10T01:50:52.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image=image_8 2022-04-10T01:50:52.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ create_image cluster2 mirror image_8 512M 2022-04-10T01:50:52.525 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:50:52.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:50:52.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:50:52.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:50:52.526 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_8 2022-04-10T01:50:52.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:50:52.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local size=128 2022-04-10T01:50:52.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:50:52.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:50:52.527 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:50:52.528 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:50:52.528 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' -n 512M ']' 2022-04-10T01:50:52.528 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ size=512M 2022-04-10T01:50:52.528 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ shift 2022-04-10T01:50:52.528 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_8 2022-04-10T01:50:52.601 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ enable_mirror cluster2 mirror image_8 2022-04-10T01:50:52.601 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:50:52.601 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:50:52.601 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_8 2022-04-10T01:50:52.602 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local mode=snapshot 2022-04-10T01:50:52.602 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:50:52.602 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:50:52.602 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:50:52.603 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:50:52.603 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2022-04-10T01:50:52.857 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~292s 2022-04-10T01:50:54.039 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:Mirroring enabled 2022-04-10T01:50:54.175 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2022-04-10T01:50:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:rbd image 'image_8': 2022-04-10T01:50:54.792 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: size 512 MiB in 128 objects 2022-04-10T01:50:54.792 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: order 22 (4 MiB objects) 2022-04-10T01:50:54.792 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: snapshot_count: 1 2022-04-10T01:50:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: id: 19053c6adc20 2022-04-10T01:50:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: block_name_prefix: rbd_data.19053c6adc20 2022-04-10T01:50:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: format: 2 2022-04-10T01:50:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: features: layering, exclusive-lock 2022-04-10T01:50:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: op_features: 2022-04-10T01:50:54.794 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: flags: 2022-04-10T01:50:54.794 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: create_timestamp: Sun Apr 10 01:50:52 2022 2022-04-10T01:50:54.794 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: access_timestamp: Sun Apr 10 01:50:52 2022 2022-04-10T01:50:54.794 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: modify_timestamp: Sun Apr 10 01:50:52 2022 2022-04-10T01:50:54.795 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring state: enabled 2022-04-10T01:50:54.795 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring mode: snapshot 2022-04-10T01:50:54.795 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring global id: 3de081c4-30a5-4c3f-90b8-36e7e5b90493 2022-04-10T01:50:54.795 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: mirroring primary: true 2022-04-10T01:50:54.926 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ stress_write_image cluster2 mirror image_8 2022-04-10T01:50:54.926 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:50:54.926 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:50:54.927 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_8 2022-04-10T01:50:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2022-04-10T01:50:54.947 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local duration=11 2022-04-10T01:50:54.947 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:50:54.947 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:50:54.948 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:50:54.948 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:50:54.948 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T01:50:54.948 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2022-04-10T01:50:55.697 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:100 IOs, 61628416 bytes 2022-04-10T01:50:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:200 IOs, 99551232 bytes 2022-04-10T01:50:55.971 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:300 IOs, 133666816 bytes 2022-04-10T01:50:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:400 IOs, 193239040 bytes 2022-04-10T01:50:56.250 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:500 IOs, 259083264 bytes 2022-04-10T01:50:56.440 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:600 IOs, 375658496 bytes 2022-04-10T01:50:56.546 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:700 IOs, 433001472 bytes 2022-04-10T01:50:56.690 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:800 IOs, 498166784 bytes 2022-04-10T01:50:56.796 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:900 IOs, 524758016 bytes 2022-04-10T01:50:56.947 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1000 IOs, 600929280 bytes 2022-04-10T01:50:57.068 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1100 IOs, 647173120 bytes 2022-04-10T01:50:57.206 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1200 IOs, 723454976 bytes 2022-04-10T01:50:57.309 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1300 IOs, 758984704 bytes 2022-04-10T01:50:57.426 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1400 IOs, 822240256 bytes 2022-04-10T01:50:57.505 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1500 IOs, 826932224 bytes 2022-04-10T01:50:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1600 IOs, 876228608 bytes 2022-04-10T01:50:57.736 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1700 IOs, 934620160 bytes 2022-04-10T01:50:57.824 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1800 IOs, 956491776 bytes 2022-04-10T01:50:57.914 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:1900 IOs, 979761152 bytes 2022-04-10T01:50:57.982 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2000 IOs, 980706304 bytes 2022-04-10T01:50:58.074 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2100 IOs, 1010816000 bytes 2022-04-10T01:50:58.181 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2200 IOs, 1053199360 bytes 2022-04-10T01:50:58.261 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~297s 2022-04-10T01:50:58.269 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2300 IOs, 1078841344 bytes 2022-04-10T01:50:58.404 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2400 IOs, 1163901952 bytes 2022-04-10T01:50:58.565 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2500 IOs, 1252557824 bytes 2022-04-10T01:50:58.643 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2600 IOs, 1270301696 bytes 2022-04-10T01:50:58.805 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2700 IOs, 1373267968 bytes 2022-04-10T01:50:58.948 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2800 IOs, 1461484544 bytes 2022-04-10T01:50:59.066 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:2900 IOs, 1517900800 bytes 2022-04-10T01:50:59.161 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3000 IOs, 1544845312 bytes 2022-04-10T01:50:59.325 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3100 IOs, 1662276608 bytes 2022-04-10T01:50:59.440 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3200 IOs, 1711533056 bytes 2022-04-10T01:50:59.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3300 IOs, 1739159552 bytes 2022-04-10T01:50:59.660 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3400 IOs, 1815050240 bytes 2022-04-10T01:50:59.760 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3500 IOs, 1847591936 bytes 2022-04-10T01:50:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3600 IOs, 1848188928 bytes 2022-04-10T01:50:59.955 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3700 IOs, 1899806720 bytes 2022-04-10T01:51:00.031 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3800 IOs, 1900434432 bytes 2022-04-10T01:51:00.120 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:3900 IOs, 1921062912 bytes 2022-04-10T01:51:00.222 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4000 IOs, 1961351168 bytes 2022-04-10T01:51:00.329 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4100 IOs, 2003437568 bytes 2022-04-10T01:51:00.457 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4200 IOs, 2073479168 bytes 2022-04-10T01:51:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4300 IOs, 2105992192 bytes 2022-04-10T01:51:00.646 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4400 IOs, 2131969024 bytes 2022-04-10T01:51:00.808 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4500 IOs, 2248055808 bytes 2022-04-10T01:51:00.918 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4600 IOs, 2299318272 bytes 2022-04-10T01:51:01.003 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4700 IOs, 2318873600 bytes 2022-04-10T01:51:01.129 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4800 IOs, 2372208640 bytes 2022-04-10T01:51:01.229 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:4900 IOs, 2412925952 bytes 2022-04-10T01:51:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5000 IOs, 2513892352 bytes 2022-04-10T01:51:01.518 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5100 IOs, 2586174464 bytes 2022-04-10T01:51:01.604 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5200 IOs, 2601461760 bytes 2022-04-10T01:51:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5300 IOs, 2649328640 bytes 2022-04-10T01:51:01.828 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5400 IOs, 2693622784 bytes 2022-04-10T01:51:01.943 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5500 IOs, 2758690816 bytes 2022-04-10T01:51:02.086 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5600 IOs, 2839448576 bytes 2022-04-10T01:51:02.228 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5700 IOs, 2916017152 bytes 2022-04-10T01:51:02.352 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5800 IOs, 2978583552 bytes 2022-04-10T01:51:02.452 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:5900 IOs, 3019280384 bytes 2022-04-10T01:51:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6000 IOs, 3081031680 bytes 2022-04-10T01:51:02.732 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6100 IOs, 3171241984 bytes 2022-04-10T01:51:02.854 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6200 IOs, 3232633856 bytes 2022-04-10T01:51:02.966 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6300 IOs, 3295032320 bytes 2022-04-10T01:51:03.111 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6400 IOs, 3351904256 bytes 2022-04-10T01:51:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6500 IOs, 3362717696 bytes 2022-04-10T01:51:03.312 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6600 IOs, 3413878784 bytes 2022-04-10T01:51:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6700 IOs, 3436123136 bytes 2022-04-10T01:51:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6800 IOs, 3454670848 bytes 2022-04-10T01:51:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:6900 IOs, 3483984896 bytes 2022-04-10T01:51:03.665 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~303s 2022-04-10T01:51:03.666 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2022-04-10T01:51:03.673 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7000 IOs, 3514218496 bytes 2022-04-10T01:51:03.769 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7100 IOs, 3549824000 bytes 2022-04-10T01:51:03.891 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7200 IOs, 3606902784 bytes 2022-04-10T01:51:03.990 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7300 IOs, 3646519296 bytes 2022-04-10T01:51:04.069 INFO:tasks.ceph.cluster2.osd.1:Sent signal 15 2022-04-10T01:51:04.070 INFO:tasks.ceph.cluster2.osd.2:Sent signal 15 2022-04-10T01:51:04.070 INFO:tasks.ceph.cluster2.mon.a:Sent signal 15 2022-04-10T01:51:04.071 INFO:tasks.ceph.cluster2.mgr.x:Sent signal 15 2022-04-10T01:51:04.071 INFO:tasks.ceph.cluster2.osd.1.smithi043.stderr:2022-04-10T01:51:04.067+0000 7f181325f700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 (PID: 34545) UID: 0 2022-04-10T01:51:04.072 INFO:tasks.ceph.cluster2.osd.1.smithi043.stderr:2022-04-10T01:51:04.067+0000 7f181325f700 -1 osd.1 511 *** Got signal Terminated *** 2022-04-10T01:51:04.072 INFO:tasks.ceph.cluster2.osd.1.smithi043.stderr:2022-04-10T01:51:04.067+0000 7f181325f700 -1 osd.1 511 *** Immediate shutdown (osd_fast_shutdown=true) *** 2022-04-10T01:51:04.073 INFO:tasks.ceph.cluster2.osd.2.smithi043.stderr:2022-04-10T01:51:04.068+0000 7f8bf24e4700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 (PID: 34550) UID: 0 2022-04-10T01:51:04.073 INFO:tasks.ceph.cluster2.osd.2.smithi043.stderr:2022-04-10T01:51:04.068+0000 7f8bf24e4700 -1 osd.2 511 *** Got signal Terminated *** 2022-04-10T01:51:04.073 INFO:tasks.ceph.cluster2.osd.2.smithi043.stderr:2022-04-10T01:51:04.068+0000 7f8bf24e4700 -1 osd.2 511 *** Immediate shutdown (osd_fast_shutdown=true) *** 2022-04-10T01:51:04.074 INFO:tasks.ceph.cluster2.mon.a.smithi043.stderr:2022-04-10T01:51:04.068+0000 7f00f00fe700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster cluster2 -i a (PID: 34205) UID: 0 2022-04-10T01:51:04.074 INFO:tasks.ceph.cluster2.mon.a.smithi043.stderr:2022-04-10T01:51:04.068+0000 7f00f00fe700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2022-04-10T01:51:04.074 INFO:tasks.ceph.cluster2.mgr.x.smithi005.stderr:2022-04-10T01:51:04.070+0000 7f75f8e93700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster cluster2 -i x (PID: 39017) UID: 0 2022-04-10T01:51:04.075 INFO:tasks.ceph.cluster2.mgr.x.smithi005.stderr:2022-04-10T01:51:04.070+0000 7f75f8e93700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2022-04-10T01:51:04.097 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7400 IOs, 3674167296 bytes 2022-04-10T01:51:04.176 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7500 IOs, 3674786816 bytes 2022-04-10T01:51:04.296 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7600 IOs, 3725031424 bytes 2022-04-10T01:51:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7700 IOs, 3795073024 bytes 2022-04-10T01:51:04.561 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7800 IOs, 3859146752 bytes 2022-04-10T01:51:04.682 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:7900 IOs, 3929197568 bytes 2022-04-10T01:51:04.766 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8000 IOs, 3936703488 bytes 2022-04-10T01:51:04.867 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8100 IOs, 3975818240 bytes 2022-04-10T01:51:04.974 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8200 IOs, 4015640576 bytes 2022-04-10T01:51:05.061 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8300 IOs, 4033501184 bytes 2022-04-10T01:51:05.164 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8400 IOs, 4058262528 bytes 2022-04-10T01:51:05.275 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8500 IOs, 4097020928 bytes 2022-04-10T01:51:05.400 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8600 IOs, 4157760512 bytes 2022-04-10T01:51:05.496 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8700 IOs, 4184902656 bytes 2022-04-10T01:51:05.619 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8800 IOs, 4240178176 bytes 2022-04-10T01:51:05.720 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:8900 IOs, 4270460928 bytes 2022-04-10T01:51:05.802 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:9000 IOs, 4279274496 bytes 2022-04-10T01:51:05.901 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:9100 IOs, 4313830400 bytes 2022-04-10T01:51:05.971 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ error_code=124 2022-04-10T01:51:05.972 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set -e 2022-04-10T01:51:05.972 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 124 -eq 124 ']' 2022-04-10T01:51:05.972 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:51:05.972 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ wait_for_image_present cluster1 mirror image_8 present 2022-04-10T01:51:05.973 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:51:05.973 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:51:05.973 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_8 2022-04-10T01:51:05.973 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local state=present 2022-04-10T01:51:05.973 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id= 2022-04-10T01:51:05.974 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local s 2022-04-10T01:51:05.974 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:51:05.974 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:51:05.974 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:51:05.975 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:51:05.975 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n '' 2022-04-10T01:51:05.975 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_8 2022-04-10T01:51:05.975 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:51:05.975 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:51:05.976 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_8 2022-04-10T01:51:05.976 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:51:05.976 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:51:05.976 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:51:05.977 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:51:05.977 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2022-04-10T01:51:05.977 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:51:06.009 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ image_id=1133285ba150 2022-04-10T01:51:06.010 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 64 64 2m 2022-04-10T01:51:06.010 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ sleep 0.1 2022-04-10T01:51:06.112 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test_image_present cluster1 mirror image_8 present 1133285ba150 2022-04-10T01:51:06.112 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:51:06.112 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:51:06.113 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_8 2022-04-10T01:51:06.113 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local test_state=present 2022-04-10T01:51:06.113 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image_id=1133285ba150 2022-04-10T01:51:06.113 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_state=deleted 2022-04-10T01:51:06.114 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local current_image_id 2022-04-10T01:51:06.114 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster1 2022-04-10T01:51:06.114 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster1 2022-04-10T01:51:06.114 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:51:06.114 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:51:06.115 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ get_image_id cluster1 mirror image_8 2022-04-10T01:51:06.115 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:51:06.115 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local pool=mirror 2022-04-10T01:51:06.115 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local image=image_8 2022-04-10T01:51:06.115 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ can_execute cluster1 2022-04-10T01:51:06.116 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster=cluster1 2022-04-10T01:51:06.116 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' cluster1 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:51:06.116 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ return 0 2022-04-10T01:51:06.116 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2022-04-10T01:51:06.116 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2022-04-10T01:51:06.151 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_image_id=1133285ba150 2022-04-10T01:51:06.151 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -n 1133285ba150 2022-04-10T01:51:06.151 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test -z 1133285ba150 -o 1133285ba150 = 1133285ba150 2022-04-10T01:51:06.152 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ current_state=present 2022-04-10T01:51:06.152 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ test present = present 2022-04-10T01:51:06.152 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:51:06.152 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ disable_mirror cluster2 mirror image_8 2022-04-10T01:51:06.152 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:51:06.153 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local pool=mirror 2022-04-10T01:51:06.153 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local image=image_8 2022-04-10T01:51:06.153 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ can_execute cluster2 2022-04-10T01:51:06.153 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster=cluster2 2022-04-10T01:51:06.153 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' cluster2 = cluster3 -a 0 -ne 1 ']' 2022-04-10T01:51:06.154 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ return 0 2022-04-10T01:51:06.154 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_8 2022-04-10T01:51:52.154 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7047266513439265) > weight (0.25) 2022-04-10T01:51:52.154 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5126585147990615) > weight (0.25) 2022-04-10T01:51:52.155 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.6960381570730735) > weight (0.25) 2022-04-10T01:51:52.155 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.4418291508438853) > weight (0.25) 2022-04-10T01:51:52.155 INFO:tasks.rbd_mirror_thrash:waiting for 109 secs before thrashing 2022-04-10T01:52:01.318 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2022-04-10T01:52:01.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2022-04-10T01:52:01.319 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2022-04-10T01:52:01.320 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr:2022-04-10T01:52:01.318+0000 7f9ed3803700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 51939) UID: 1000 2022-04-10T01:52:09.320 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2022-04-10T01:52:09.321 INFO:teuthology.orchestra.run:waiting for 600 2022-04-10T01:53:41.156 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5121820662986628) > weight (0.25) 2022-04-10T01:53:41.157 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5901271666590627) > weight (0.25) 2022-04-10T01:53:41.158 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.32331087318293483) > weight (0.25) 2022-04-10T01:53:41.158 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2022-04-10T01:53:41.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2022-04-10T01:53:41.159 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2022-04-10T01:53:41.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr:2022-04-10T01:53:41.158+0000 7fa3b8608700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 67904) UID: 1000 2022-04-10T01:53:43.160 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2022-04-10T01:53:43.312 INFO:teuthology.orchestra.run:waiting for 600 2022-04-10T02:02:13.448 INFO:tasks.rbd_mirror_thrash:Failed to stop cluster2.client.mirror.0 2022-04-10T02:02:13.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 6 2022-04-10T02:02:13.452 ERROR:tasks.rbd_mirror_thrash:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_9b3739e792ef182b79445b926c7b7da461c05f52/qa/tasks/rbd_mirror_thrash.py", line 84, in _run self.do_thrash() File "/home/teuthworker/src/git.ceph.com_ceph-c_9b3739e792ef182b79445b926c7b7da461c05f52/qa/tasks/rbd_mirror_thrash.py", line 156, in do_thrash run.wait([daemon.proc], timeout=600) File "/home/teuthworker/src/git.ceph.com_git_teuthology_a347b2ac5400a43ba76a678399b1deda7dae1a6c/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_git_teuthology_a347b2ac5400a43ba76a678399b1deda7dae1a6c/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (100) after waiting for 600 seconds 2022-04-10T02:02:13.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr:*** Caught signal (Aborted) ** 2022-04-10T02:02:13.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: in thread 7f9edc9b36c0 thread_name:rbd-mirror 2022-04-10T02:02:13.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: ceph version 17.0.0-11492-g9b3739e7 (9b3739e792ef182b79445b926c7b7da461c05f52) quincy (dev) 2022-04-10T02:02:13.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f9eda33ace0] 2022-04-10T02:02:13.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: 2: pthread_cond_wait() 2022-04-10T02:02:13.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: 3: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f9ed91438f0] 2022-04-10T02:02:13.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: 4: /lib64/librados.so.2(+0xc7c63) [0x7f9edc471c63] 2022-04-10T02:02:13.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: 5: /lib64/librados.so.2(+0xba5a4) [0x7f9edc4645a4] 2022-04-10T02:02:13.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: 6: (rbd::mirror::ClusterWatcher::read_pool_peers(std::map >, std::less, std::allocator > > > >*)+0x38) [0x55f19ae844d8] 2022-04-10T02:02:13.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: 7: (rbd::mirror::ClusterWatcher::refresh_pools()+0x22d) [0x55f19ae869cd] 2022-04-10T02:02:13.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: 8: (rbd::mirror::Mirror::run()+0x1d6) [0x55f19ae43796] 2022-04-10T02:02:13.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: 9: main() 2022-04-10T02:02:13.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: 10: __libc_start_main() 2022-04-10T02:02:13.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: 11: _start() 2022-04-10T02:02:13.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr:2022-04-10T02:02:13.451+0000 7f9edc9b36c0 -1 *** Caught signal (Aborted) ** 2022-04-10T02:02:13.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: in thread 7f9edc9b36c0 thread_name:rbd-mirror 2022-04-10T02:02:13.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: 2022-04-10T02:02:13.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: ceph version 17.0.0-11492-g9b3739e7 (9b3739e792ef182b79445b926c7b7da461c05f52) quincy (dev) 2022-04-10T02:02:13.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f9eda33ace0] 2022-04-10T02:02:13.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: 2: pthread_cond_wait() 2022-04-10T02:02:13.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: 3: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f9ed91438f0] 2022-04-10T02:02:13.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: 4: /lib64/librados.so.2(+0xc7c63) [0x7f9edc471c63] 2022-04-10T02:02:13.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: 5: /lib64/librados.so.2(+0xba5a4) [0x7f9edc4645a4] 2022-04-10T02:02:13.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: 6: (rbd::mirror::ClusterWatcher::read_pool_peers(std::map >, std::less, std::allocator > > > >*)+0x38) [0x55f19ae844d8] 2022-04-10T02:02:13.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: 7: (rbd::mirror::ClusterWatcher::refresh_pools()+0x22d) [0x55f19ae869cd] 2022-04-10T02:02:13.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: 8: (rbd::mirror::Mirror::run()+0x1d6) [0x55f19ae43796] 2022-04-10T02:02:13.460 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: 9: main() 2022-04-10T02:02:13.460 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: 10: __libc_start_main() 2022-04-10T02:02:13.460 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: 11: _start() 2022-04-10T02:02:13.460 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-04-10T02:02:13.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: 2022-04-10T02:02:13.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: -9827> 2022-04-10T01:44:36.935+0000 7f9ecc7f5700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f1a10a09c0 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T02:02:13.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: -9824> 2022-04-10T01:44:36.935+0000 7f9ed3002700 -1 rbd::mirror::ImageReplayer: 0x55f1a0b2b800 [3/59134803-ff09-429b-9467-2cdbc3d6d23c] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T02:02:13.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: -9466> 2022-04-10T01:44:46.724+0000 7f9ecbff4700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f1a10a0b60 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T02:02:13.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: -9463> 2022-04-10T01:44:46.724+0000 7f9ed3002700 -1 rbd::mirror::ImageReplayer: 0x55f19e64dc00 [3/be4c9569-5fe3-47a0-a0a1-84d380d56af1] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T02:02:13.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: -8610> 2022-04-10T01:44:52.748+0000 7f9ecbff4700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f1a10b1ba0 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T02:02:13.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: -8607> 2022-04-10T01:44:52.748+0000 7f9ed2801700 -1 rbd::mirror::ImageReplayer: 0x55f1a0b2ac00 [3/a8a88faf-cd63-4557-9afd-ca8d6505edb6] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T02:02:13.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: -8455> 2022-04-10T01:44:54.735+0000 7f9ecbff4700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f1a10a16c0 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T02:02:13.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: -8452> 2022-04-10T01:44:54.735+0000 7f9ed3002700 -1 rbd::mirror::ImageReplayer: 0x55f1a0b29600 [3/ef9f36c8-5f7f-4c73-8f03-3933b20566c7] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T02:02:13.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: -8274> 2022-04-10T01:44:58.760+0000 7f9ecbff4700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f1a10b1d40 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T02:02:13.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: -8271> 2022-04-10T01:44:58.760+0000 7f9ed2801700 -1 rbd::mirror::ImageReplayer: 0x55f1a0b29800 [3/e100174d-382f-42fc-9a30-2ef4065b1b6a] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T02:02:13.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: -7661> 2022-04-10T01:45:20.831+0000 7f9ecc7f5700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f1a10a01a0 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T02:02:13.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: -7658> 2022-04-10T01:45:20.831+0000 7f9ed2801700 -1 rbd::mirror::ImageReplayer: 0x55f1a0b2ba00 [3/e2bbf2e7-46b6-4dd8-b6e1-9dea8cf3b0b4] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T02:02:13.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: -7476> 2022-04-10T01:45:21.384+0000 7f9ecbff4700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f1a10a1ba0 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T02:02:13.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: -7454> 2022-04-10T01:45:21.384+0000 7f9ed3002700 -1 rbd::mirror::ImageReplayer: 0x55f1a0b2ba00 [3/e2bbf2e7-46b6-4dd8-b6e1-9dea8cf3b0b4] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T02:02:13.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: -7047> 2022-04-10T01:45:28.866+0000 7f9ecbff4700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f1a10a1d40 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T02:02:13.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: -7044> 2022-04-10T01:45:28.866+0000 7f9ed3002700 -1 rbd::mirror::ImageReplayer: 0x55f1a0b29400 [3/bcd924f1-5832-47a7-92fc-1dbfdf17dc65] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T02:02:13.467 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: -2692> 2022-04-10T01:50:21.384+0000 7f9ecc7f5700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f19f2d2820 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T02:02:13.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: -2689> 2022-04-10T01:50:21.384+0000 7f9ed3002700 -1 rbd::mirror::ImageReplayer: 0x55f1a0b2ae00 [3/d4207619-dba4-47d6-8fe0-da5f8bdc5e87] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T02:02:13.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: -1623> 2022-04-10T01:52:01.318+0000 7f9ed3803700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 51939) UID: 1000 2022-04-10T02:02:13.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: 0> 2022-04-10T02:02:13.451+0000 7f9edc9b36c0 -1 *** Caught signal (Aborted) ** 2022-04-10T02:02:13.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: in thread 7f9edc9b36c0 thread_name:rbd-mirror 2022-04-10T02:02:13.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: 2022-04-10T02:02:13.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: ceph version 17.0.0-11492-g9b3739e7 (9b3739e792ef182b79445b926c7b7da461c05f52) quincy (dev) 2022-04-10T02:02:13.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f9eda33ace0] 2022-04-10T02:02:13.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: 2: pthread_cond_wait() 2022-04-10T02:02:13.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: 3: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f9ed91438f0] 2022-04-10T02:02:13.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: 4: /lib64/librados.so.2(+0xc7c63) [0x7f9edc471c63] 2022-04-10T02:02:13.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: 5: /lib64/librados.so.2(+0xba5a4) [0x7f9edc4645a4] 2022-04-10T02:02:13.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: 6: (rbd::mirror::ClusterWatcher::read_pool_peers(std::map >, std::less, std::allocator > > > >*)+0x38) [0x55f19ae844d8] 2022-04-10T02:02:13.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: 7: (rbd::mirror::ClusterWatcher::refresh_pools()+0x22d) [0x55f19ae869cd] 2022-04-10T02:02:13.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: 8: (rbd::mirror::Mirror::run()+0x1d6) [0x55f19ae43796] 2022-04-10T02:02:13.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: 9: main() 2022-04-10T02:02:13.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: 10: __libc_start_main() 2022-04-10T02:02:13.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: 11: _start() 2022-04-10T02:02:13.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-04-10T02:02:13.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: 2022-04-10T02:02:13.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: -9999> 2022-04-10T01:44:36.935+0000 7f9ecc7f5700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f1a10a09c0 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T02:02:13.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: -9998> 2022-04-10T01:44:36.935+0000 7f9ed3002700 -1 rbd::mirror::ImageReplayer: 0x55f1a0b2b800 [3/59134803-ff09-429b-9467-2cdbc3d6d23c] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T02:02:13.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: -9997> 2022-04-10T01:44:46.724+0000 7f9ecbff4700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f1a10a0b60 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T02:02:13.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: -9996> 2022-04-10T01:44:46.724+0000 7f9ed3002700 -1 rbd::mirror::ImageReplayer: 0x55f19e64dc00 [3/be4c9569-5fe3-47a0-a0a1-84d380d56af1] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T02:02:13.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: -9995> 2022-04-10T01:44:52.748+0000 7f9ecbff4700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f1a10b1ba0 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T02:02:13.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: -9994> 2022-04-10T01:44:52.748+0000 7f9ed2801700 -1 rbd::mirror::ImageReplayer: 0x55f1a0b2ac00 [3/a8a88faf-cd63-4557-9afd-ca8d6505edb6] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T02:02:13.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: -9993> 2022-04-10T01:44:54.735+0000 7f9ecbff4700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f1a10a16c0 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T02:02:13.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: -9992> 2022-04-10T01:44:54.735+0000 7f9ed3002700 -1 rbd::mirror::ImageReplayer: 0x55f1a0b29600 [3/ef9f36c8-5f7f-4c73-8f03-3933b20566c7] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T02:02:13.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: -9991> 2022-04-10T01:44:58.760+0000 7f9ecbff4700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f1a10b1d40 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T02:02:13.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: -9990> 2022-04-10T01:44:58.760+0000 7f9ed2801700 -1 rbd::mirror::ImageReplayer: 0x55f1a0b29800 [3/e100174d-382f-42fc-9a30-2ef4065b1b6a] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T02:02:13.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: -9989> 2022-04-10T01:45:20.831+0000 7f9ecc7f5700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f1a10a01a0 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T02:02:13.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: -9988> 2022-04-10T01:45:20.831+0000 7f9ed2801700 -1 rbd::mirror::ImageReplayer: 0x55f1a0b2ba00 [3/e2bbf2e7-46b6-4dd8-b6e1-9dea8cf3b0b4] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T02:02:13.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: -9987> 2022-04-10T01:45:21.384+0000 7f9ecbff4700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f1a10a1ba0 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T02:02:13.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: -9986> 2022-04-10T01:45:21.384+0000 7f9ed3002700 -1 rbd::mirror::ImageReplayer: 0x55f1a0b2ba00 [3/e2bbf2e7-46b6-4dd8-b6e1-9dea8cf3b0b4] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T02:02:13.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: -9985> 2022-04-10T01:45:28.866+0000 7f9ecbff4700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f1a10a1d40 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T02:02:13.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: -9984> 2022-04-10T01:45:28.866+0000 7f9ed3002700 -1 rbd::mirror::ImageReplayer: 0x55f1a0b29400 [3/bcd924f1-5832-47a7-92fc-1dbfdf17dc65] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T02:02:13.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: -9983> 2022-04-10T01:50:21.384+0000 7f9ecc7f5700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55f19f2d2820 handle_prepare_local_image: error preparing local image for replay(85) Interrupted system call should be restarted 2022-04-10T02:02:13.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: -9982> 2022-04-10T01:50:21.384+0000 7f9ed3002700 -1 rbd::mirror::ImageReplayer: 0x55f1a0b2ae00 [3/d4207619-dba4-47d6-8fe0-da5f8bdc5e87] operator(): start failed: (85) Interrupted system call should be restarted 2022-04-10T02:02:13.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: -9981> 2022-04-10T01:52:01.318+0000 7f9ed3803700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 51939) UID: 1000 2022-04-10T02:02:13.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: -9980> 2022-04-10T02:02:13.451+0000 7f9edc9b36c0 -1 *** Caught signal (Aborted) ** 2022-04-10T02:02:13.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: in thread 7f9edc9b36c0 thread_name:rbd-mirror 2022-04-10T02:02:13.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: 2022-04-10T02:02:13.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: ceph version 17.0.0-11492-g9b3739e7 (9b3739e792ef182b79445b926c7b7da461c05f52) quincy (dev) 2022-04-10T02:02:13.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f9eda33ace0] 2022-04-10T02:02:13.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: 2: pthread_cond_wait() 2022-04-10T02:02:13.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: 3: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f9ed91438f0] 2022-04-10T02:02:13.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: 4: /lib64/librados.so.2(+0xc7c63) [0x7f9edc471c63] 2022-04-10T02:02:13.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: 5: /lib64/librados.so.2(+0xba5a4) [0x7f9edc4645a4] 2022-04-10T02:02:13.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: 6: (rbd::mirror::ClusterWatcher::read_pool_peers(std::map >, std::less, std::allocator > > > >*)+0x38) [0x55f19ae844d8] 2022-04-10T02:02:13.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: 7: (rbd::mirror::ClusterWatcher::refresh_pools()+0x22d) [0x55f19ae869cd] 2022-04-10T02:02:13.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: 8: (rbd::mirror::Mirror::run()+0x1d6) [0x55f19ae43796] 2022-04-10T02:02:13.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: 9: main() 2022-04-10T02:02:13.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: 10: __libc_start_main() 2022-04-10T02:02:13.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: 11: _start() 2022-04-10T02:02:13.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-04-10T02:02:13.484 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr: 2022-04-10T02:02:13.853 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi043.stderr:daemon-helper: command crashed with signal 6 2022-04-10T02:03:47.338 INFO:tasks.rbd_mirror_thrash:Failed to stop cluster1.client.mirror.3 2022-04-10T02:03:47.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 6 2022-04-10T02:03:47.340 ERROR:tasks.rbd_mirror_thrash:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_9b3739e792ef182b79445b926c7b7da461c05f52/qa/tasks/rbd_mirror_thrash.py", line 84, in _run self.do_thrash() File "/home/teuthworker/src/git.ceph.com_ceph-c_9b3739e792ef182b79445b926c7b7da461c05f52/qa/tasks/rbd_mirror_thrash.py", line 156, in do_thrash run.wait([daemon.proc], timeout=600) File "/home/teuthworker/src/git.ceph.com_git_teuthology_a347b2ac5400a43ba76a678399b1deda7dae1a6c/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_git_teuthology_a347b2ac5400a43ba76a678399b1deda7dae1a6c/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (100) after waiting for 600 seconds 2022-04-10T02:03:47.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr:*** Caught signal (Aborted) ** 2022-04-10T02:03:47.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: in thread 7fa3c17b86c0 thread_name:rbd-mirror 2022-04-10T02:03:47.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: ceph version 17.0.0-11492-g9b3739e7 (9b3739e792ef182b79445b926c7b7da461c05f52) quincy (dev) 2022-04-10T02:03:47.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7fa3bf13fce0] 2022-04-10T02:03:47.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 2: pthread_cond_wait() 2022-04-10T02:03:47.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 3: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7fa3bdf488f0] 2022-04-10T02:03:47.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 4: (C_SaferCond::wait()+0x63) [0x562b64dcb6c3] 2022-04-10T02:03:47.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 5: (rbd::mirror::PoolReplayer::remove_peer(std::_Rb_tree_iterator > >&, C_GatherBase*)+0x9a) [0x562b64dd15da] 2022-04-10T02:03:47.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 6: (rbd::mirror::PoolReplayer::shut_down()+0x10e) [0x562b64dd81de] 2022-04-10T02:03:47.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 7: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x10) [0x562b64dd8480] 2022-04-10T02:03:47.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 8: (std::_Rb_tree, std::default_delete > > >, std::_Select1st, std::default_delete > > > >, std::less, std::allocator, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node, std::default_delete > > > >*)+0x3a) [0x562b64dc6d1a] 2022-04-10T02:03:47.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 9: (std::_Rb_tree, std::default_delete > > >, std::_Select1st, std::default_delete > > > >, std::less, std::allocator, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node, std::default_delete > > > >*)+0x25) [0x562b64dc6d05] 2022-04-10T02:03:47.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 10: (rbd::mirror::Mirror::~Mirror()+0xc5) [0x562b64dc1945] 2022-04-10T02:03:47.345 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 11: main() 2022-04-10T02:03:47.345 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 12: __libc_start_main() 2022-04-10T02:03:47.345 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 13: _start() 2022-04-10T02:03:47.345 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr:2022-04-10T02:03:47.339+0000 7fa3c17b86c0 -1 *** Caught signal (Aborted) ** 2022-04-10T02:03:47.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: in thread 7fa3c17b86c0 thread_name:rbd-mirror 2022-04-10T02:03:47.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 2022-04-10T02:03:47.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: ceph version 17.0.0-11492-g9b3739e7 (9b3739e792ef182b79445b926c7b7da461c05f52) quincy (dev) 2022-04-10T02:03:47.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7fa3bf13fce0] 2022-04-10T02:03:47.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 2: pthread_cond_wait() 2022-04-10T02:03:47.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 3: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7fa3bdf488f0] 2022-04-10T02:03:47.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 4: (C_SaferCond::wait()+0x63) [0x562b64dcb6c3] 2022-04-10T02:03:47.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 5: (rbd::mirror::PoolReplayer::remove_peer(std::_Rb_tree_iterator > >&, C_GatherBase*)+0x9a) [0x562b64dd15da] 2022-04-10T02:03:47.348 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 6: (rbd::mirror::PoolReplayer::shut_down()+0x10e) [0x562b64dd81de] 2022-04-10T02:03:47.348 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 7: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x10) [0x562b64dd8480] 2022-04-10T02:03:47.348 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 8: (std::_Rb_tree, std::default_delete > > >, std::_Select1st, std::default_delete > > > >, std::less, std::allocator, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node, std::default_delete > > > >*)+0x3a) [0x562b64dc6d1a] 2022-04-10T02:03:47.348 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 9: (std::_Rb_tree, std::default_delete > > >, std::_Select1st, std::default_delete > > > >, std::less, std::allocator, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node, std::default_delete > > > >*)+0x25) [0x562b64dc6d05] 2022-04-10T02:03:47.349 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 10: (rbd::mirror::Mirror::~Mirror()+0xc5) [0x562b64dc1945] 2022-04-10T02:03:47.349 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 11: main() 2022-04-10T02:03:47.349 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 12: __libc_start_main() 2022-04-10T02:03:47.349 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 13: _start() 2022-04-10T02:03:47.349 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-04-10T02:03:47.350 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 2022-04-10T02:03:47.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: -2350> 2022-04-10T01:53:41.158+0000 7fa3b8608700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 67904) UID: 1000 2022-04-10T02:03:47.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 0> 2022-04-10T02:03:47.339+0000 7fa3c17b86c0 -1 *** Caught signal (Aborted) ** 2022-04-10T02:03:47.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: in thread 7fa3c17b86c0 thread_name:rbd-mirror 2022-04-10T02:03:47.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 2022-04-10T02:03:47.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: ceph version 17.0.0-11492-g9b3739e7 (9b3739e792ef182b79445b926c7b7da461c05f52) quincy (dev) 2022-04-10T02:03:47.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7fa3bf13fce0] 2022-04-10T02:03:47.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 2: pthread_cond_wait() 2022-04-10T02:03:47.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 3: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7fa3bdf488f0] 2022-04-10T02:03:47.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 4: (C_SaferCond::wait()+0x63) [0x562b64dcb6c3] 2022-04-10T02:03:47.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 5: (rbd::mirror::PoolReplayer::remove_peer(std::_Rb_tree_iterator > >&, C_GatherBase*)+0x9a) [0x562b64dd15da] 2022-04-10T02:03:47.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 6: (rbd::mirror::PoolReplayer::shut_down()+0x10e) [0x562b64dd81de] 2022-04-10T02:03:47.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 7: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x10) [0x562b64dd8480] 2022-04-10T02:03:47.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 8: (std::_Rb_tree, std::default_delete > > >, std::_Select1st, std::default_delete > > > >, std::less, std::allocator, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node, std::default_delete > > > >*)+0x3a) [0x562b64dc6d1a] 2022-04-10T02:03:47.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 9: (std::_Rb_tree, std::default_delete > > >, std::_Select1st, std::default_delete > > > >, std::less, std::allocator, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node, std::default_delete > > > >*)+0x25) [0x562b64dc6d05] 2022-04-10T02:03:47.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 10: (rbd::mirror::Mirror::~Mirror()+0xc5) [0x562b64dc1945] 2022-04-10T02:03:47.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 11: main() 2022-04-10T02:03:47.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 12: __libc_start_main() 2022-04-10T02:03:47.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 13: _start() 2022-04-10T02:03:47.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-04-10T02:03:47.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 2022-04-10T02:03:47.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: -7889> 2022-04-10T01:53:41.158+0000 7fa3b8608700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 67904) UID: 1000 2022-04-10T02:03:47.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: -7888> 2022-04-10T02:03:47.339+0000 7fa3c17b86c0 -1 *** Caught signal (Aborted) ** 2022-04-10T02:03:47.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: in thread 7fa3c17b86c0 thread_name:rbd-mirror 2022-04-10T02:03:47.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 2022-04-10T02:03:47.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: ceph version 17.0.0-11492-g9b3739e7 (9b3739e792ef182b79445b926c7b7da461c05f52) quincy (dev) 2022-04-10T02:03:47.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7fa3bf13fce0] 2022-04-10T02:03:47.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 2: pthread_cond_wait() 2022-04-10T02:03:47.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 3: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7fa3bdf488f0] 2022-04-10T02:03:47.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 4: (C_SaferCond::wait()+0x63) [0x562b64dcb6c3] 2022-04-10T02:03:47.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 5: (rbd::mirror::PoolReplayer::remove_peer(std::_Rb_tree_iterator > >&, C_GatherBase*)+0x9a) [0x562b64dd15da] 2022-04-10T02:03:47.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 6: (rbd::mirror::PoolReplayer::shut_down()+0x10e) [0x562b64dd81de] 2022-04-10T02:03:47.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 7: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x10) [0x562b64dd8480] 2022-04-10T02:03:47.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 8: (std::_Rb_tree, std::default_delete > > >, std::_Select1st, std::default_delete > > > >, std::less, std::allocator, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node, std::default_delete > > > >*)+0x3a) [0x562b64dc6d1a] 2022-04-10T02:03:47.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 9: (std::_Rb_tree, std::default_delete > > >, std::_Select1st, std::default_delete > > > >, std::less, std::allocator, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node, std::default_delete > > > >*)+0x25) [0x562b64dc6d05] 2022-04-10T02:03:47.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 10: (rbd::mirror::Mirror::~Mirror()+0xc5) [0x562b64dc1945] 2022-04-10T02:03:47.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 11: main() 2022-04-10T02:03:47.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 12: __libc_start_main() 2022-04-10T02:03:47.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 13: _start() 2022-04-10T02:03:47.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-04-10T02:03:47.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr: 2022-04-10T02:03:47.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi043.stderr:daemon-helper: command crashed with signal 6 2022-04-10T02:03:50.991 INFO:tasks.daemonwatchdog.daemon_watchdog:thrasher.rbd_mirror.[cluster1] failed 2022-04-10T02:03:50.991 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2022-04-10T02:03:51.495 INFO:tasks.ceph.cluster1.osd.0:Sent signal 15 2022-04-10T02:03:51.496 INFO:tasks.ceph.cluster1.osd.1:Sent signal 15 2022-04-10T02:03:51.496 INFO:tasks.ceph.cluster1.osd.2:Sent signal 15 2022-04-10T02:03:51.497 INFO:tasks.ceph.cluster1.mon.a:Sent signal 15 2022-04-10T02:03:51.497 INFO:tasks.ceph.cluster1.mgr.x:Sent signal 15 2022-04-10T02:03:51.498 INFO:tasks.ceph.cluster1.osd.0.smithi005.stderr:2022-04-10T02:03:51.495+0000 7fb19e5b5700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 (PID: 33831) UID: 0 2022-04-10T02:03:51.498 INFO:tasks.ceph.cluster1.osd.0.smithi005.stderr:2022-04-10T02:03:51.495+0000 7fb19e5b5700 -1 osd.0 445 *** Got signal Terminated *** 2022-04-10T02:03:51.499 INFO:tasks.ceph.cluster1.osd.1.smithi005.stderr:2022-04-10T02:03:51.495+0000 7f858f305700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 (PID: 33837) UID: 0 2022-04-10T02:03:51.499 INFO:tasks.ceph.cluster1.osd.1.smithi005.stderr:2022-04-10T02:03:51.495+0000 7f858f305700 -1 osd.1 445 *** Got signal Terminated *** 2022-04-10T02:03:51.499 INFO:tasks.ceph.cluster1.osd.1.smithi005.stderr:2022-04-10T02:03:51.495+0000 7f858f305700 -1 osd.1 445 *** Immediate shutdown (osd_fast_shutdown=true) *** 2022-04-10T02:03:51.500 INFO:tasks.ceph.cluster1.osd.2.smithi005.stderr:2022-04-10T02:03:51.495+0000 7f0c73027700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 (PID: 33841) UID: 0 2022-04-10T02:03:51.500 INFO:tasks.ceph.cluster1.osd.2.smithi005.stderr:2022-04-10T02:03:51.495+0000 7f0c73027700 -1 osd.2 445 *** Got signal Terminated *** 2022-04-10T02:03:51.500 INFO:tasks.ceph.cluster1.osd.2.smithi005.stderr:2022-04-10T02:03:51.495+0000 7f0c73027700 -1 osd.2 445 *** Immediate shutdown (osd_fast_shutdown=true) *** 2022-04-10T02:03:51.501 INFO:tasks.ceph.cluster1.osd.0.smithi005.stderr:2022-04-10T02:03:51.495+0000 7fb19e5b5700 -1 osd.0 445 *** Immediate shutdown (osd_fast_shutdown=true) *** 2022-04-10T02:03:51.502 INFO:tasks.ceph.cluster1.mon.a.smithi005.stderr:2022-04-10T02:03:51.495+0000 7f4b2fc53700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster cluster1 -i a (PID: 33458) UID: 0 2022-04-10T02:03:51.502 INFO:tasks.ceph.cluster1.mon.a.smithi005.stderr:2022-04-10T02:03:51.495+0000 7f4b2fc53700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2022-04-10T02:03:51.502 INFO:tasks.ceph.cluster1.mgr.x.smithi005.stderr:2022-04-10T02:03:51.496+0000 7fa275d43700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster cluster1 -i x (PID: 33462) UID: 0 2022-04-10T02:03:51.503 INFO:tasks.ceph.cluster1.mgr.x.smithi005.stderr:2022-04-10T02:03:51.496+0000 7fa275d43700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2022-04-10T02:41:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: couldn't connect to the cluster! 2022-04-10T02:41:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ cleanup 110 2022-04-10T02:41:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local error_code=110 2022-04-10T02:41:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ set +e 2022-04-10T02:41:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ '[' 110 -ne 0 ']' 2022-04-10T02:41:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ================================= 2022-04-10T02:41:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:================================= 2022-04-10T02:41:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:FIRST STATUS 2022-04-10T02:41:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:================================= 2022-04-10T02:41:06.183 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo FIRST STATUS 2022-04-10T02:41:06.183 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo ================================= 2022-04-10T02:41:06.183 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ status 2022-04-10T02:41:06.183 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ local cluster daemon image_pool image_ns image 2022-04-10T02:41:06.184 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for cluster in ${CLUSTERS} 2022-04-10T02:41:06.184 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:cluster1 status 2022-04-10T02:41:06.185 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo 'cluster1 status' 2022-04-10T02:41:06.185 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ CEPH_ARGS= 2022-04-10T02:41:06.185 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ ceph --cluster cluster1 -s 2022-04-10T02:46:06.283 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:2022-04-10T02:46:06.280+0000 7faf7e26f700 0 monclient(hunting): authenticate timed out after 300 2022-04-10T02:51:06.283 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:2022-04-10T02:51:06.282+0000 7faf7e26f700 0 monclient(hunting): authenticate timed out after 300 2022-04-10T02:56:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:2022-04-10T02:56:06.282+0000 7faf7e26f700 0 monclient(hunting): authenticate timed out after 300 2022-04-10T03:01:06.285 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:2022-04-10T03:01:06.283+0000 7faf7e26f700 0 monclient(hunting): authenticate timed out after 300 2022-04-10T03:06:06.285 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:2022-04-10T03:06:06.283+0000 7faf7e26f700 0 monclient(hunting): authenticate timed out after 300 2022-04-10T03:11:06.285 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:2022-04-10T03:11:06.283+0000 7faf7e26f700 0 monclient(hunting): authenticate timed out after 300 2022-04-10T03:16:06.288 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:2022-04-10T03:16:06.284+0000 7faf7e26f700 0 monclient(hunting): authenticate timed out after 300 2022-04-10T03:21:06.289 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:2022-04-10T03:21:06.284+0000 7faf7e26f700 0 monclient(hunting): authenticate timed out after 300 2022-04-10T03:26:06.290 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:2022-04-10T03:26:06.285+0000 7faf7e26f700 0 monclient(hunting): authenticate timed out after 300 2022-04-10T03:31:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:2022-04-10T03:31:06.286+0000 7faf7e26f700 0 monclient(hunting): authenticate timed out after 300 2022-04-10T03:31:06.292 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2022-04-10T03:31:06.293 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ CEPH_ARGS= 2022-04-10T03:31:06.293 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ ceph --cluster cluster1 service dump 2022-04-10T03:36:06.398 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:2022-04-10T03:36:06.394+0000 7f6f7d046700 0 monclient(hunting): authenticate timed out after 300 2022-04-10T03:41:06.398 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:2022-04-10T03:41:06.394+0000 7f6f7d046700 0 monclient(hunting): authenticate timed out after 300 2022-04-10T03:46:06.399 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:2022-04-10T03:46:06.395+0000 7f6f7d046700 0 monclient(hunting): authenticate timed out after 300 2022-04-10T03:51:06.399 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:2022-04-10T03:51:06.396+0000 7f6f7d046700 0 monclient(hunting): authenticate timed out after 300 2022-04-10T03:56:06.401 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:2022-04-10T03:56:06.396+0000 7f6f7d046700 0 monclient(hunting): authenticate timed out after 300 2022-04-10T04:01:06.401 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:2022-04-10T04:01:06.396+0000 7f6f7d046700 0 monclient(hunting): authenticate timed out after 300 2022-04-10T04:06:06.401 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:2022-04-10T04:06:06.398+0000 7f6f7d046700 0 monclient(hunting): authenticate timed out after 300 2022-04-10T04:11:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:2022-04-10T04:11:06.398+0000 7f6f7d046700 0 monclient(hunting): authenticate timed out after 300 2022-04-10T04:16:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:2022-04-10T04:16:06.398+0000 7f6f7d046700 0 monclient(hunting): authenticate timed out after 300 2022-04-10T04:21:06.404 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:2022-04-10T04:21:06.398+0000 7f6f7d046700 0 monclient(hunting): authenticate timed out after 300 2022-04-10T04:21:06.405 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2022-04-10T04:21:06.405 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ CEPH_ARGS= 2022-04-10T04:21:06.405 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ ceph --cluster cluster1 service status 2022-04-10T04:26:06.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:2022-04-10T04:26:06.505+0000 7fe6e2a0e700 0 monclient(hunting): authenticate timed out after 300 2022-04-10T04:31:06.509 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:2022-04-10T04:31:06.505+0000 7fe6e2a0e700 0 monclient(hunting): authenticate timed out after 300 2022-04-10T04:36:06.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:2022-04-10T04:36:06.506+0000 7fe6e2a0e700 0 monclient(hunting): authenticate timed out after 300 2022-04-10T04:41:06.510 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:2022-04-10T04:41:06.507+0000 7fe6e2a0e700 0 monclient(hunting): authenticate timed out after 300 2022-04-10T04:46:06.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:2022-04-10T04:46:06.507+0000 7fe6e2a0e700 0 monclient(hunting): authenticate timed out after 300 2022-04-10T04:51:06.511 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:2022-04-10T04:51:06.507+0000 7fe6e2a0e700 0 monclient(hunting): authenticate timed out after 300 2022-04-10T04:56:06.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:2022-04-10T04:56:06.508+0000 7fe6e2a0e700 0 monclient(hunting): authenticate timed out after 300 2022-04-10T05:01:06.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:2022-04-10T05:01:06.509+0000 7fe6e2a0e700 0 monclient(hunting): authenticate timed out after 300 2022-04-10T05:06:06.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:2022-04-10T05:06:06.509+0000 7fe6e2a0e700 0 monclient(hunting): authenticate timed out after 300 2022-04-10T05:11:06.513 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:2022-04-10T05:11:06.510+0000 7fe6e2a0e700 0 monclient(hunting): authenticate timed out after 300 2022-04-10T05:11:06.534 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2022-04-10T05:11:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: 2022-04-10T05:11:06.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo 2022-04-10T05:11:06.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for image_pool in ${POOL} ${PARENT_POOL} 2022-04-10T05:11:06.536 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2022-04-10T05:11:06.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:cluster1 mirror images 2022-04-10T05:11:06.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo 'cluster1 mirror images' 2022-04-10T05:11:06.537 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' ls -l 2022-04-10T06:01:06.552 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: couldn't connect to the cluster! 2022-04-10T06:01:06.553 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: listing images failed: (110) Connection timed out 2022-04-10T06:01:06.554 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo 2022-04-10T06:01:06.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: 2022-04-10T06:01:06.555 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:cluster1 mirror mirror pool info 2022-04-10T06:01:06.556 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo 'cluster1 mirror mirror pool info' 2022-04-10T06:01:06.556 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' mirror pool info 2022-04-10T06:51:06.587 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:rbd: couldn't connect to the cluster! 2022-04-10T06:51:06.615 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo 2022-04-10T06:51:06.616 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout: 2022-04-10T06:51:06.617 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:cluster1 mirror mirror pool status 2022-04-10T06:51:06.618 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ echo 'cluster1 mirror mirror pool status' 2022-04-10T06:51:06.618 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ CEPH_ARGS= 2022-04-10T06:51:06.619 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' mirror pool status --verbose 2022-04-10T07:17:42.685 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:Terminated 2022-04-10T07:17:42.716 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ cleanup 143 2022-04-10T07:17:42.716 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local error_code=143 2022-04-10T07:17:42.716 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ set +e 2022-04-10T07:17:42.717 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ '[' 143 -ne 0 ']' 2022-04-10T07:17:42.717 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo ================================= 2022-04-10T07:17:42.718 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:================================= 2022-04-10T07:17:42.718 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:FIRST STATUS 2022-04-10T07:17:42.718 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:================================= 2022-04-10T07:17:42.719 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo FIRST STATUS 2022-04-10T07:17:42.719 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo ================================= 2022-04-10T07:17:42.719 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ status 2022-04-10T07:17:42.719 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ local cluster daemon image_pool image_ns image 2022-04-10T07:17:42.720 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ for cluster in ${CLUSTERS} 2022-04-10T07:17:42.720 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ echo 'cluster1 status' 2022-04-10T07:17:42.720 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stdout:cluster1 status 2022-04-10T07:17:42.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ CEPH_ARGS= 2022-04-10T07:17:42.721 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:++ ceph --cluster cluster1 -s 2022-04-10T07:22:42.787 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:2022-04-10T07:22:42.784+0000 7fd2439a6700 0 monclient(hunting): authenticate timed out after 300 2022-04-10T07:27:42.788 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:2022-04-10T07:27:42.785+0000 7fd2439a6700 0 monclient(hunting): authenticate timed out after 300 2022-04-10T07:32:42.788 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:2022-04-10T07:32:42.785+0000 7fd2439a6700 0 monclient(hunting): authenticate timed out after 300 2022-04-10T07:37:42.789 INFO:tasks.workunit.cluster1.client.mirror.smithi043.stderr:2022-04-10T07:37:42.786+0000 7fd2439a6700 0 monclient(hunting): authenticate timed out after 300 2022-04-10T07:38:34.404 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2022-04-10T07:38:34.409 DEBUG:teuthology.task.console_log:Killing console logger for smithi005 2022-04-10T07:38:34.410 DEBUG:teuthology.task.console_log:Killing console logger for smithi043 2022-04-10T07:38:34.411 DEBUG:teuthology.exit:Finished running handlers